Method for obtaining a composite image using rotationally symmetrical wide-angle lenses, imaging system for same, and CMOS image sensor for image-processing using hardware

ABSTRACT

The present invention provides a method for obtaining a composite image, which performs a mathematically correct image-processing on images obtained by wide-angle lenses which are rotationally symmetrical with regard to an optical axis, to achieve a desirable projection system. The present invention also provides a variety of imaging systems using the method. Further, the present invention provides a CMOS image sensor which has an accurate arrangement of pixels to perform image processing using hardware without the need for image processing by software.

TECHNICAL FIELD

The present invention generally relates to mathematically precise imageprocessing methods of extracting complex images from images acquiredusing a camera equipped with a wide-angle lens that is rotationallysymmetric about an optical axis and imaging systems employing thesemethods, as well as CMOS image sensors for hardwired image processing.

BACKGROUND ART

Panoramic camera, which captures the 360° view of scenic places such astourist resorts, is an example of a panoramic imaging system. Panoramicimaging system is an imaging system that captures the views one couldget by making one complete turn-around from a given spot. On the otherhand, an omnidirectional imaging system captures the views of everypossible direction from a given position. Omnidirectional imaging systemprovides a view that a person could observe from a given position byturning around as well as looking up and down. In a mathematicalterminology, the solid angle of the region that can be captured by theimaging system is 4π steradian.

There have been a lot of studies and developments of panoramic andomnidirectional imaging systems not only in the traditional areas suchas photographing buildings, nature scenes, and heavenly bodies, but alsoin security/surveillance systems using CCD (charge-coupled device) orCMOS (complementary metal-oxide-semiconductor) cameras, virtual touringof real estates, hotels and tourist resort, and navigational aids formobile robots and unmanned aerial vehicles (UAV).

One method of obtaining a panoramic image is to employ a fisheye lenswith a wide field of view (FOV). For example, the stars in the entiresky and the horizon can be captured in a single image by pointing acamera equipped with a fisheye lens with 180° FOV toward the zenith(i.e., the optical axis of the camera is aligned perpendicular to theground plane). On this reason, fisheye lenses have been often referredto as “all-sky lenses”. Particularly, a high-end fisheye lens by Nikon,namely, 6 mm f/5.6 Fisheye-Nikkor, has a FOV of 220°. Therefore, acamera equipped with this lens can capture a portion of the backside ofthe camera as well as the front side of the camera. Then, panoramicimage can be obtained from thus obtained fisheye image after properimage processing.

In many cases, imaging systems are installed on vertical walls. Imagingsystems installed on outside walls of a building for the purpose ofmonitoring the surroundings, or a rear view camera for monitoring thebackside of a passenger car are such examples. In such cases, it israther inefficient if the horizontal field of view is significantlylarger than 180°. This is because a wall, which is not needed to bemonitored, takes up a large space in the monitor screen. Pixels arewasted in this case, and the screen appears dull. Therefore, ahorizontal FOV around 180° is more appropriate for such cases.Nevertheless, a fisheye lens with 180° FOV is not desirable for suchapplications. This is because the barrel distortion, which accompanies afisheye lens, evokes psychological discomfort and thus abhorred by theconsumer.

References 1 and 2 provide fundamental technologies of extracting animage having a particular viewpoint or projection scheme from an imagehaving a different viewpoint or projection scheme. Specifically,reference 2 provides an example of a cubic panorama. In short, a cubicpanorama is a special technique of illustration wherein the observer isassumed to be located at the very center of an imaginary cubic room madeof glass, and the outside view from the center of the glass room isdirectly transcribed on the region of the glass wall where the rayvector from the object to the observer meets the glass wall. However,the environment was not a real environment captured by an optical lens,but a computer-created imaginary environment captured with an imaginarydistortion-free pinhole camera.

From another point of view, all animals and plants including human arebound on the surface of the earth due to the gravitational pull, andmost of the events, which need attention or cautionary measure, takeplace near the horizon. Therefore, even though it is necessary tomonitor every 360° direction on the horizon, it is not as important tomonitor high along the vertical direction, for example, as high as tothe zenith or deep down to the nadir. Distortion is unavoidable if wewant to describe the scene of every 360° direction on a two-dimensionalplane. Similar difficulty exists in the cartography where geography onEarth, which is a structure on the surface of a sphere, needs to bemapped on a planar two-dimensional atlas.

All the animals, plants and inanimate objects such as buildings on theEarth are all under the influence of gravity, and the direction of thegravitational force is the up-right direction or the vertical direction.Among all the distortions, the distortion that appears most unnatural tothe human is the distortion where vertical lines appear as curved lines.Therefore, even if other kinds of distortions are present, it isimportant to make sure that such a distortion is absent. In general, aground plane is perpendicular to the direction of the gravitationalforce, but it not so on a slanted ground. Therefore, in a strict senseof the words, a reference has to be made with respect to the horizontalplane, and a vertical direction is a direction perpendicular to thehorizontal plane.

Described in reference 3 are the well-known map projection schemes amongthe diverse map projection schemes such as equi-rectangular projection,Mercator projection and cylindrical projection schemes, and reference 4provides a brief history of diverse map projection schemes. Among these,the equi-rectangular projection scheme is the projection scheme mostfamiliar to us when we describe the geography on the Earth, or when wedraw the celestial sphere in order to make a map of the constellation.

Referring to FIG. 1, if we assume the surface of the Earth or thecelestial sphere is a spherical surface with a radius S, then anarbitrary point Q on the Earth's surface with respect to the center N ofthe Earth has a longitude ψ and a latitude δ. On the other hand, FIG. 2is a schematic diagram of a planar map of the Earth's surface or thecelestial sphere drawn according to the equi-rectangular projectionscheme. A point Q on the Earth's surface having a longitude V and alatitude δ has a corresponding point P″ on the planar map(234) drawnaccording to the equi-rectangular projection scheme. The rectangularcoordinate of this corresponding point is given as (x″, y″).Furthermore, the reference point on the equator having a longitude 0°and a latitude 0° has a corresponding point ◯″ on the planar map, andthis corresponding point ◯″ is the origin of the rectangular coordinatesystem. Here, according to the equi-rectangular projection scheme, asame interval in the longitude (i.e., a same angular distance along theequatorial line) corresponds to a same lateral interval on the planarmap. In other words, the lateral coordinate x″ on the planar map(234) isproportional to the longitude.x″=cψ  [Equation 1]

Here, c is proportionality constant. Also, the longitudinal coordinatey″ is proportional to the latitude, and has the same proportionalityconstant as the lateral coordinate.y″=cδ  [Equation 2]

Such an equi-rectangular projection scheme appears as a naturalprojection scheme considering the fact that the Earth's surface is closeto a sphere. Nevertheless, it is disadvantageous in that the size of ageographical area is greatly distorted. For example, two very closepoints near the North Pole can appear as if they are on the oppositesides of the Earth in a map drawn according to the equi-rectangularprojection scheme.

On the other hand, in a map drawn according to the Mercator projectionscheme, the longitudinal coordinate is given as a complex function givenin Eq. 3.

$\begin{matrix}{y^{''} = {c\;\ln\left\{ {\tan\left( {\frac{\pi}{4} + \frac{\delta}{2}} \right)} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

On the other hand, FIG. 3 is a conceptual drawing illustrating acylindrical projection scheme or a panoramic perspective. In acylindrical projection scheme, an imaginary observer is located at thecenter N of a celestial sphere(331) with a radius S, and it is desiredto make a map of the celestial sphere centered on the observer, the mapcovering most of the region, but excluding the zenith and the nadir. Inother words, the span of the longitude must be 360° ranging from −180°to +180°, but the range of the latitude can be narrower including theequator within its span. Specifically, the span of the latitude can beassumed as ranging from −Δ to +Δ, and here, Δ must be smaller than 90°.

In this projection scheme, an imaginary cylindrical plane(334) isassumed which contacts the celestial sphere(331) at the equator(303).Then, for a point Q(ψ, δ) on the celestial sphere having a givenlongitude ψ and a latitude δ, a line segment connecting the center N ofthe celestial sphere and the point Q is extended until it meets the saidcylindrical plane. This intersection point is designated as P(ψ, δ). Inthis manner, a corresponding point P on the cylindrical plane(334) canbe obtained for every point Q on the celestial sphere(331) within thesaid latitude range. Then, a map having a cylindrical projection schemeis obtained when the cylindrical plane is cut and flattened out.Therefore, the lateral coordinate x″ of the point P on the flattened-outcylindrical plane is given by Eq. 4, and the longitudinal coordinate y″is given by Eq. 5.x″=Sψ  [Equation 4]y″=S tan δ  [Equation 5]

Such a cylindrical projection scheme is used in panoramic cameras thatproduce panoramic images by rotating in the horizontal plane.Especially, if the lens mounted on the rotating panoramic camera is adistortion-free rectilinear lens, then the resulting panoramic imageexactly follows a cylindrical projection scheme. In principle, such acylindrical projection scheme is the most accurate panoramic projectionscheme. However, the panoramic image appears unnatural when thelatitudinal range is large, and thus it is not widely used in practice.

References 5 and 6 provide an example of a fisheye lens with 190° FOV,and reference 7 provides various examples of wide-angle lenses includingdioptric and catadioptric fisheye lenses with stereographic projectionschemes.

On the other hand, reference 8 provides various examples of obtainingpanoramic images following cylindrical projection schemes,equi-rectangular projection schemes, or Mercator projection schemes fromimages acquired using rotationally symmetric wide-angle lenses includingfisheye lenses. Referring to FIG. 4 through FIG. 12, most of theexamples provided in the said references can be summarized as follows.

FIG. 4 is a conceptual drawing illustrating a real projection scheme ofa rotationally symmetric wide-angle lens(412) including a fisheye lens.Z-axis of the world coordinate system describing objects captured by thewide-angle lens coincides with the optical axis(401) of the wide-anglelens(412). An incident ray(405) having a zenith angle θ with respect tothe Z-axis is refracted by the lens(412), and as a refracted ray(406),converges toward an image point P on the focal plane(432). The distancebetween the nodal point N of the lens and the said focal plane isapproximately equal to the effective focal length of the lens. The subarea on the focal plane whereon real image points have been formed isthe image plane(433). To obtain a sharp image, the said image plane(433)must coincide with the image sensor plane(413) within the camerabody(414). Said focal plane and the said image sensor plane areperpendicular to the optical axis. The intersection point ◯ between theoptical axis(401) and the image plane(433) is hereinafter referred to asthe first intersection point. The distance between the firstintersection point and the said image point P is r.

For general wide-angle lenses, the image height r is given by Eq. 6.r=r(θ)  [Equation 6]

Here, the unit of the incidence angle θ is radian, and the abovefunction r(θ) is a monotonically increasing function of the zenith angleθ of the incident ray.

Such a real projection scheme of a lens can be experimentally measuredusing an actual lens, or can be calculated from the lens prescriptionusing dedicated lens design software such as Code V or Zemax. Forexample, the y-axis coordinate y of the image point on the focal planeby an incident ray having given horizontal and vertical incidence anglescan be calculated using a Zemax operator REAY, and the x-axis coordinatex can be similarly calculated using an operator REAX.

FIG. 5 is an imaginary interior scene produced by Professor Paul Bourkeby using a computer, and it has been assumed that an imaginary fisheyelens with 180° FOV and having an ideal equidistance projection schemehas been used to capture the scene. This image is a square image, ofwhich both the lateral and the longitudinal dimensions are 250 pixels.Therefore, the coordinate of the optical axis is (125.5, 125.5), and theimage height for an incident ray with a zenith angle of 90° is given asr′(π/2)=125.5−1=124.5. Here, r′ is not a physical distance, but an imageheight measured in pixel distance. Since this imaginary fisheye lensfollows an equidistance projection scheme, the projection scheme of thislens is given by Eq. 7.

$\begin{matrix}{{r^{\prime}(\theta)} = {{\frac{124.5}{\left( \frac{\pi}{2} \right)}\theta} = {79.26\;\theta}}} & \left\lbrack {{Equation}\mspace{14mu} 7} \right\rbrack\end{matrix}$

FIG. 6 through FIG. 8 show diverse embodiments of wide-angle lensespresented in reference 7. FIG. 6 is a dioptric (i.e., refractive)fisheye lens with a stereographic projection scheme, FIG. 7 is acatadioptric fisheye lens with a stereographic projection scheme, andFIG. 8 is a catadioptric panoramic lens with a rectilinear projectionscheme. In this manner, wide-angle lenses from the prior arts and in thecurrent invention are not limited to fisheye lenses with equidistanceprojection schemes, but encompass all kind of wide-angle lenses that arerotationally symmetric about the optical axes.

The main point of the invention in reference 8 is about providingmethods of obtaining panoramic images by applying mathematicallyaccurate image processing algorithms on images obtained usingrotationally symmetric wide-angle lenses. Referring to FIG. 9, diverseembodiments in reference 8 can be summarized as follows. FIG. 9 is aschematic diagram of a world coordinate system of prior arts.

The world coordinate system of the said invention takes the nodal pointN of a rotationally symmetric wide-angle lens as the origin, and avertical line passing through the origin as the Y-axis. Here, thevertical line is a line perpendicular to the ground plane, or moreprecisely to the horizontal plane(917). The X-axis and the Z-axis of theworld coordinate system are contained within the ground plane. Theoptical axis(901) of the said wide-angle lens generally does notcoincide with the Y-axis, and can be contained within the ground plane(i.e., parallel to the ground plane), or may not contained within theground plane. The plane(904) containing both the said Y-axis and thesaid optical axis(901) is referred to as the reference plane. Theintersection line(902) between this reference plane(904) and the groundplane(917) coincides with the Z-axis of the world coordinate system. Onthe other hand, an incident ray(905) originating from an object point Qhaving a rectangular coordinate (X, Y, Z) in the world coordinate systemhas an altitude angle δ from the ground plane, and an azimuth angle ψwith respect to the reference plane. The plane(906) containing both theY-axis and the said incident ray(905) is the incidence plane. Thehorizontal incidence angle ψ of the said incident ray with respect tothe said reference plane is given by Eq. 8.

$\begin{matrix}{\psi = {\tan^{- 1}\left( \frac{X}{Z} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack\end{matrix}$

On the other hand, the vertical incidence angle (i.e., the altitudeangle) δ subtended by the said incident ray and the X-Z plane is givenby Eq. 9.

$\begin{matrix}{\delta = {\tan^{- 1}\left( \frac{Y}{\sqrt{X^{2} + Z^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 9} \right\rbrack\end{matrix}$

The elevation angle μ of the said incident ray is given by Eq. 10,wherein χ is an arbitrary angle larger than −90° and smaller than 90°.μ=δ−χ  [Equation 10]

FIG. 10 is a schematic diagram of a panoramic imaging system, and mainlycomprised of an image acquisition means(1010), an image processingmeans(1016) and image display means(1015, 1017). The image acquisitionmeans(1010) includes a rotationally symmetric wide-angle lens(1012) anda camera body(1014) having an image sensor(1013) inside. The saidwide-angle lens can be a fisheye lens with more than 180° FOV and havingan equidistance projection scheme, but it is by no means limited to sucha fisheye lens. Rather, it can be any rotationally symmetric wide-anglelens including a catadioptric fisheye lens. Hereinafter, for the sake ofnotational simplicity, a wide-angle lens will be referred to as afisheye lens. Said camera body contains photodiodes such as CCD or CMOSsensors, and it can acquire either a still image or a movie. By the saidfisheye lens(1012), a real image of the object plane(1031) is formed onthe focal plane(1032). In order to obtain a sharp image, the imagesensor plane(1013) must coincide with the focal plane(1032). In FIG. 10,the symbol 1001 refers to the optical axis.

The real image of the objects on the object plane(1031) formed by thefisheye lens(1012) is converted by the image sensor(1013) intoelectrical signals, and displayed as an uncorrected image plane(1034) onan image display means(1015). This uncorrected image plane(1034) is adistorted image by the fisheye lens. This distorted image plane can berectified by the image processing means(1016), and then displayed as aprocessed image plane(1035) on an image display means(1017) such as acomputer monitor or a CCTV monitor. Said image processing can beaccomplished by software running on a computer, or it can be done byembedded software running on FPGA (Field Programmable Gate Array), CPLD(Complex Programmable Logic Device), ARM core processor, ASIC(Application-Specific Integrated Circuit), or DSP (Digital SignalProcessor) chips.

An arbitrary rotationally symmetric lens including a fisheye lens doesnot provide said panoramic images or distortion-free rectilinear images.Therefore, image processing stage is essential in order to obtaindesirable images. FIG. 11 is a conceptual drawing of an uncorrectedimage plane(1134) prior to an image processing stage, which correspondsto the real image on the image sensor plane(1013). If the lateraldimension of the image sensor plane(1013) is B and the longitudinaldimension is V, then the lateral dimension of the uncorrected imageplane is gB and the longitudinal dimension is gV, where g isproportionality constant.

Uncorrected image plane(1134) can be considered as an image displayed onan image display means without rectification of distortion, and is amagnified image of the real image on the image sensor plane by amagnification ratio g. For example, the image sensor plane of a ⅓-inchCCD sensor has a rectangular shape having a lateral dimension of 4.8 mm,and a longitudinal dimension of 3.6 mm. On the other hand, if themonitor is 48 cm in width and 36 cm in height, then the magnificationratio g is 100. More desirably, the side dimension of a pixel in digitalimage is considered as 1. A VGA-grade ⅓-inch CCD sensor has pixels in anarray form with 640 columns and 480 rows. Therefore, each pixel has aright rectangular shape with both width and height measuring as 4.8mm/640=7.5 μm, and in this case, the magnification ratio g is given by 1pixel/7.5 μm=133.3 pixel/mm. In recapitulation, the uncorrected imageplane(1134) is a distorted digital image obtained by converting the realimage formed on the image sensor plane into electrical signals.

The first intersection point ◯ on the image sensor plane is theintersection point between the optical axis and the image sensor plane.Therefore, a ray entered along the optical axis forms an image point onthe said first intersection point ◯. By definition, the point ◯′ on theuncorrected image plane corresponding to the first intersection point ◯on the image sensor plane—hereinafter referred to as the secondintersection point—corresponds to an image point by an incident rayentered along the optical axis.

A second rectangular coordinate systems is assumed wherein x′-axis istaken as the axis that passes through the second intersection point ◯′on the uncorrected image plane and is parallel to the sides of theuncorrected image plane along the lateral direction, and y′-axis istaken as the axis that passes through the said second intersection pointand is parallel to the sides of the uncorrected image plane along thelongitudinal direction. The positive direction of the x′-axis runs fromthe left side to the right side, and the positive direction of they′-axis runs from the top end to the bottom end. Then, the lateralcoordinate x′ of an arbitrary point on the uncorrected image plane(1134)has a minimum value x′₁=gx₁ and a maximum value x′₂=gx₂(i.e.,gx₁≦x′≦gx₂). In the same manner, the longitudinal coordinate y′ of thesaid point has a minimum value y′₁=gy₁ and a maximum value y′₂=gy₂(i.e.,gy₁≦y′≦gy₂).

FIG. 12 is a conceptual drawing of a processed image plane(1235) of theimage display means of the current invention, wherein the distortion hasbeen removed. The processed image plane(1235) has a rectangular shape,of which the lateral side measuring as W and the longitudinal sidemeasuring as H. Furthermore, a third rectangular coordinate system isassumed wherein x″-axis is parallel to the sides of the processed imageplane along the lateral direction, and y″-axis is parallel to the sidesof the processed image plane along the longitudinal direction. Thez″-axis of the third rectangular coordinate system coincides with thez-axis of the first rectangular coordinate system and the z′-axis of thesecond rectangular coordinate system. The intersection point ◯″ betweenthe said z″-axis and the processed image plane—hereinafter referred toas the third intersection point—can take an arbitrary position, and itcan even be located outside the processed image plane. Here, thepositive direction of the x″-axis runs from the left side to the rightside, and the positive direction of the y″-axis runs from the top end tothe bottom end.

The first and the second intersection points correspond to the locationof the optical axis. On the other hand, the third intersection pointcorresponds not to the location of the optical axis but to the principaldirection of vision. The principal direction of vision may coincide withthe optical axis, but it is not needed to. Principal direction of visionis the direction of the optical axis of an imaginary panoramic orrectilinear camera corresponding to the desired panoramic or rectilinearimages. Hereinafter, for the sake of notational simplicity, theprincipal direction of vision is referred to as the optical axisdirection.

The lateral coordinate x″ of a third point P″ on the processed imageplane(1235) has a minimum value x″₁ and a maximum value x″₂(i.e.,x″₁≦x″≦x″₂). By definition, the difference between the maximum lateralcoordinate and the minimum lateral coordinate is the lateral dimensionof the processed image plane(i.e., x″₂−x″₁=W). In the same manner, thelongitudinal coordinate y″ of the third point P″ has a minimum value y″₁and a maximum value y″₂(i.e., y″₁≦y″≦y″₂). By definition, the differencebetween the maximum longitudinal coordinate and the minimum longitudinalcoordinate is the longitudinal dimension of the processed imageplane(i.e., y″₂−y″₁=H).

The following table 1 summarizes corresponding variables in the objectplane, the image sensor plane, the uncorrected image plane, and theprocessed image plane.

TABLE 1 uncorrected processed object image image image Surface planesensor plane plane plane lateral dimension B gB W of the planelongitudinal V gV H dimension of the plane coordinate world the firstthe second the third system coordinate rectangular rectangularrectangular system coordinate coordinate coordinate system systemlocation of the nodal point nodal point nodal point nodal pointcoordinate origin of the lens of the lens of the lens of the lens symbolof the O O′ O″ origin coordinate axes (X, Y, Z) (x, y, z) (x′, y′, z′)(x″, y″, z″) name of the object the first the second the third objectpoint or point point point point the image point symbol of the Q P P′ P″object point or the image point two-dimensional (x, y) (x′, y′) (x″, y″)coordinate of the object point or the image point

On the other hand, if we assume the coordinate of an image point P″ onthe processed image plane(1235) corresponding to an object point with acoordinate (X, Y, Z) in the world coordinate system is (x″, y″), thenthe said image processing means process the uncorrected image plane sothat an image point corresponding to an incident ray originating fromthe said object point appears on the said screen with the coordinate(x″, y″), wherein the lateral coordinate x″ of the image point is givenby Eq. 11.x″=cψ  [Equation 11]

Here, c is proportionality constant.

Furthermore, the longitudinal coordinate y″ of the said image point isgiven by Eq. 12.y″=cF(μ)  [Equation 12]

Here, F(μ) is a monotonically increasing function passing through theorigin. In mathematical terminology, it means that Eq. 13 and Eq. 14 aresatisfied.

$\begin{matrix}{{F(0)} = 0} & \left\lbrack {{Equation}\mspace{14mu} 13} \right\rbrack \\{\frac{\partial{F(\mu)}}{\partial\mu} > 0} & \left\lbrack {{Equation}\mspace{14mu} 14} \right\rbrack\end{matrix}$

The above function F(μ) can take an arbitrary form, but the mostdesirable forms are given by Eq. 15 through Eq. 18.

$\begin{matrix}{{F(\mu)} = {\tan\;\mu}} & \left\lbrack {{Equation}\mspace{14mu} 15} \right\rbrack \\{{F(\mu)} = \frac{\tan\;\mu}{\cos\;\chi}} & \left\lbrack {{Equation}\mspace{14mu} 16} \right\rbrack \\{{F(\mu)} = \mu} & \left\lbrack {{Equation}\mspace{14mu} 17} \right\rbrack \\{{F(\mu)} = {\ln\left\{ {\tan\left( {\frac{\mu}{2} + \frac{\pi}{4}} \right)} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 18} \right\rbrack\end{matrix}$

FIG. 13 is a schematic diagram for understanding the field of view (FOV)and the projection scheme of a panoramic imaging system according to anembodiment of a prior art. A panoramic imaging system of the currentembodiment is assumed as attached on a vertical wall(1330), which isperpendicular to the ground plane. The wall coincides with the X-Yplane, and the Y-axis runs from the ground plane(i.e., X-Z plane) to thezenith. The origin of the coordinate system is located at the nodalpoint N of the lens, and the optical axis(1301) of the lens coincideswith the Z-axis. In a rigorous sense, the direction of the optical axisis the direction of the negative Z-axis of the world coordinate system.This is because, by the notational convention of imaging optics, thedirection from the object (or, an object point) to the image plane(or,an image point) is the positive direction. Despite this fact, we willdescribe the optical axis as coinciding with the Z-axis of the worldcoordinate system for the sake of simplicity in argument. The presentinvention is not an invention about designing a lens but an inventionemploying a lens, and in the viewpoint of a lens user, it makes easierto understand the optical axis as in the current embodiment of thepresent invention.

The image sensor plane(1313) is a plane having a rectangular shape andperpendicular to the optical axis, whereof the lateral dimension is B,and the longitudinal dimension is V. In the current embodiment, theX-axis of the world coordinate system is parallel to the x-axis of thefirst rectangular coordinate system, and points in the same direction.On the other hand, the Y-axis of the world coordinate system is parallelto the y-axis of the first rectangular coordinate system, but thedirection of the Y-axis is the exact opposite of the direction of they-axis. Therefore, in FIG. 13, the positive direction of the x-axis ofthe first rectangular coordinate system runs from the left to the right,and the positive direction of the y-axis runs from the top to thebottom.

The intersection point ◯ between the z-axis of the first rectangularcoordinate system and the sensor plane(1313)—in other words, the firstintersection point—is not generally located at the center of the sensorplane(1313), and it can even be located outside the sensor plane. Such acase can happen when the center of the image sensor is moved away fromthe center position of the lens—i.e., the optical axis—on purpose inorder to obtain an asymmetrical vertical or horizontal field of view.

A panoramic camera with a cylindrical projection scheme follows arectilinear projection scheme in the vertical direction, and anequidistance projection scheme in the horizontal direction. Such aprojection scheme corresponds to assuming a hemi-cylindrical objectplane(1331) with a radius S and having the Y-axis as the rotationalsymmetry axis. The image of an arbitrary point Q on the objectplane(1331)—hereinafter referred to as an object point—appears as animage point P on the said sensor plane(1213). According to the desirableprojection scheme of the current embodiment, the image of an object onthe hemi-cylindrical object plane(1331) is captured on the sensorplane(1313) with its vertical proportions preserved, and the lateralcoordinate x of the image point is proportional to the horizontal arclength of the corresponding object point on the said object plane. Theimage points on the image sensor plane by all the object points on theobject plane(1331) collectively form a real image.

FIG. 14 shows the cross-sectional view of the object plane in FIG. 13 inX-Z plane, and FIG. 15 shows the cross-sectional view of the objectplane in FIG. 13 in Y-Z plane. From FIG. 13 through FIG. 15, thefollowing Eq. 19 can be obtained.

$\begin{matrix}{A = {\frac{H}{{\tan\;\delta_{2}} - {\tan\;\delta_{1}}} = {\frac{y^{''}}{\tan\;\delta} = {\frac{y_{2}^{''}}{\tan\;\delta_{2}} = {\frac{y_{1}^{''}}{\tan\;\delta_{1}} = {\frac{W}{\Delta\psi} = {\frac{x^{''}}{\psi} = {\frac{x_{1}^{''}}{\psi_{1}} = \frac{x_{2}^{''}}{\psi_{2}}}}}}}}}} & \left\lbrack {{Equation}\mspace{14mu} 19} \right\rbrack\end{matrix}$

Therefore, when setting-up the size and the FOV of a desirable processedimage plane, it must be ensured that Eq. 19 is satisfied.

If the processed image plane in FIG. 12 satisfies the said projectionscheme, then the horizontal incidence angle of an incident raycorresponding to a lateral coordinate x″ of a third point P″ on the saidprocessed image plane is given by Eq. 20.

$\begin{matrix}{\psi = {{\frac{\Delta\;\psi}{W}x^{''}} = \frac{x^{''}}{A}}} & \left\lbrack {{Equation}\mspace{14mu} 20} \right\rbrack\end{matrix}$

Likewise, the vertical incidence angle of an incident ray correspondingto the third point having a longitudinal coordinate y″ is, from Eq. 19,given as Eq. 21.

$\begin{matrix}{\delta = {\tan^{- 1}\left( \frac{y^{''}}{A} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 21} \right\rbrack\end{matrix}$

Therefore, the signal value of a third point on the processed imageplane having an ideal projection scheme must be given as the signalvalue of an image point on the image sensor plane formed by an incidentray originating from an object point on the object plane having ahorizontal incidence angle (i.e., the longitude) given by Eq. 20 and avertical incidence angle (i.e., the latitude) given by Eq. 21.

According to a prior art, a panoramic image having an ideal projectionscheme can be obtained as follows from a fisheye image having adistortion. First, according to the user's need, the size (W, H) of thepanoramic image and the location of the third intersection point ◯″ aredetermined. The said third intersection point can be located outside thesaid processed image plane. In other words, the range (x″₁, x″₂) of thelateral coordinate and the range (y″₁, y″₂) of the longitudinalcoordinate on the processed image plane can take arbitrary real numbers.Also, the horizontal field of view Δψ of this panoramic image (in otherwords, the processed image plane) is determined. Then, the horizontalincidence angle γ and the vertical incidence angle δ of an incident raycorresponding to the rectangular coordinate (x″, y″) of a third point onthe panoramic image can be obtained using Eq. 20 and Eq. 21. Next, thezenith angle θ and the azimuth angle φ of an incident ray correspondingto these horizontal and vertical incidence angles are obtained using Eq.22 and Eq. 23.

$\begin{matrix}{\phi = {\tan^{- 1}\left( \frac{\tan\;\delta}{\sin\;\psi} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 22} \right\rbrack \\{\theta = {\cos^{- 1}\left( {\cos\;{\delta cos}\;\psi} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 23} \right\rbrack\end{matrix}$

Next, the image height r corresponding to the zenith angle θ of theincident ray is obtained using Eq. 6. Then, using this image height r,the magnification ratio g and the azimuth angle φ of the incident ray,the two-dimensional rectangular coordinate (x′, y′) of the image pointon the uncorrected image plane can be obtained as in Eq. 24 and Eq. 25.x′=gr(θ)cos φ  [Equation 24]y′=gr(θ)sin φ  [Equation 25]

In this procedure, the coordinate of the second intersection point onthe uncorrected image plane, or equivalently the coordinate of the firstintersection point on the image sensor plane, has to be accuratelydetermined. Such a location of the intersection point can be easilyfound using various methods including image processing method. Sincesuch techniques are well known to the people in this field, they willnot be described in this document. Finally, the video signal (i.e., RGBsignal) value of an image point by the fisheye lens having thisrectangular coordinate is substituted as the video signal value for animage point on the panoramic image having a rectangular coordinate (x″,y″). A panoramic image having an ideal projection scheme can be obtainedby image processing for all the image points on the processed imageplane by the above-described method.

However, in reality, a complication arises due to the fact that all theimage sensors and display devices are digital devices. Processed imageplane has pixels in the form of a two-dimensional array having J_(max)columns in the lateral direction and I_(max) lows in the longitudinaldirection. Although, in general, each pixel has a square shape with boththe lateral dimension and the longitudinal dimension measuring as p, thelateral and the longitudinal dimensions of a pixel are considered as 1in the image processing field. To designate a particular pixel P″, thelow number I and the column number J are used.

There is an image point—i.e., the first point—on the image sensor planecorresponding to a pixel P″ on the said processed image plane. Thehorizontal incidence angle of an incident ray in the world coordinatesystem forming an image at this first point can be written asψ_(I,J)≡ψ(I, J). Also, the vertical incidence angle can be written asδ_(I,J)≡δ(I, J). Incidentally, the location of this first point does notgenerally coincide with the exact location of any one pixel.

Here, if the said processed image plane is a panoramic image, then asgiven by Eq. 26, the horizontal incidence angle must be a sole functionof the lateral pixel coordinates J.ψ_(I,J)=ψ_(J)≡ψ(J)  [Equation 26]

Likewise, the vertical incidence angle must be a sole function of thelongitudinal pixel coordinates I.δ_(I,J)=δ_(I)≡δ(I)  [Equation 27]

Compared with the previous image processing methods, image processingmethods for digitized images must follow the following set ofprocedures. First, the real projection scheme of the wide-angle lensthat is meant to be used for image acquisition is obtained either byexperiment or based on an accurate lens design prescription. Herein,when an incident ray having a zenith angle θ with respect to the opticalaxis forms a sharp image point on the image sensor plane by the imageforming properties of the lens, the real projection scheme of the lensrefers to the distance r from the intersection point ◯ between the saidimage sensor plane and the optical axis to the said image point obtainedas a function of the zenith angle θ of the incident ray.r=r(θ)  [Equation 28]

Said function is a monotonically increasing function of the zenith angleθ. Next, the location of the optical axis on the uncorrected imageplane, in other words, the location of the second intersection point ◯′on the uncorrected image plane corresponding to the first intersectionpoint ◯ on the image sensor plane is obtained. The pixel coordinate ofthis second intersection point is assumed as (K_(o), L_(o)). In additionto this, the magnification ratio g of the pixel distance r′ on theuncorrected image plane over the real image height r on the image sensorplane is obtained. This magnification ratio g is given by Eq. 29.

$\begin{matrix}{g = \frac{r^{\prime}}{r}} & \left\lbrack {{Equation}\mspace{14mu} 29} \right\rbrack\end{matrix}$

Once such a series of preparatory stages have been completed, then acamera mounted with the said fisheye lens is installed with its opticalaxis aligned parallel to the ground plane, and a raw image(i.e., anuncorrected image plane) is acquired. Next, a desirable size of theprocessed image plane and the location (I_(o), J_(o)) of the thirdintersection point is determined, and then the horizontal incidenceangle ψ_(J) given by Eq. 30 and the vertical incidence angle δ_(I) givenby Eq. 31 are computed for all the pixels (I, J) on the said processedimage plane.

$\begin{matrix}{\psi_{J} = {\frac{\psi_{J\;\max} - \psi_{1}}{J_{\max} - 1}\left( {J - J_{o}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 30} \right\rbrack \\{\delta_{I} = {\tan^{- 1}\left\{ {\frac{\psi_{Jmax} - \psi_{1}}{J_{\max} - 1}\left( {I - I_{o}} \right)} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 31} \right\rbrack\end{matrix}$

From these horizontal and vertical incidence angles, the zenith angleθ_(I,J) and the azimuth angle φ_(I,J) of the incident ray in the firstrectangular coordinate system are obtained using Eq. 32 and Eq. 33.

$\begin{matrix}{\theta_{I,J} = {\cos^{- 1}\left( {\cos\;\delta_{I}\cos\;\psi_{J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 32} \right\rbrack \\{\phi_{I,J} = {\tan^{- 1}\left( \frac{\tan\;\delta_{I}}{\sin\;\psi_{J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 33} \right\rbrack\end{matrix}$

Next, the image height r_(I,J) on the image sensor plane is obtainedusing Eq. 34 and Eq. 28.r _(I,J) =r(θ_(I,J))  [Equation 34]

Next, using the location (K_(o), L_(o)) of the second intersection pointon the uncorrected image plane and the magnification ratio g, thelocation of the second point on the uncorrected image plane is obtainedusing Eq. 35 and Eq. 36.x′ _(I,J) =L _(o) +gr _(I,J) cos φ_(I,J)  [Equation 35]y′ _(I,J) =K _(o) +gr _(I,J) sin φ_(I,J)  [Equation 36]

The location of the said second point does not exactly coincide with thelocation of any one pixel. Therefore, (x′_(I,J),x′_(I,J)) can beconsidered as the coordinate of a virtual pixel on the uncorrected imageplane corresponding to the third point on the processed image plane, andthey are real numbers in general.

Since the said second point does not coincide with any one pixel, anappropriate interpolation method must be used for image processing. FIG.16 is a panoramic image following a cylindrical projection scheme thathas been extracted from the image in FIG. 5, of which the lateral andthe longitudinal dimensions are both 250 pixels, and the thirdintersection point is located at the center of the processed imageplane. Furthermore, the horizontal FOV of the processed image plane is180° (i.e., π). As can be seen from FIG. 16, all the vertical lines inthe three walls, namely the front, the left, and the right walls in FIG.5 appear as straight lines in FIG. 16.

On the other hand, FIG. 17 is an exemplary image of an interior scene,which has been acquired by aligning the optical axis of a fisheye lenswith 190° FOV described in references 5 and 6 parallel to the groundplane. The real projection scheme of this fisheye lens is described indetail in the said references. On the other hand, FIG. 18 is a panoramicimage having a cylindrical projection scheme extracted from the fisheyeimage in FIG. 17. Here, the width:height ratio of the processed imageplane is 16:9, the position of the third intersection point coincideswith the center of the processed image plane, and the horizontal FOV isset as 190°. As can be seen from FIG. 18, all the vertical lines arecaptured as vertical lines and all the objects appear natural. Slighterrors are due to the error in aligning the optical axis parallel to theground plane, and the error in experimentally determining the positionof the optical axis on the uncorrected image plane.

Inventions in reference 8 provide mathematically accurate imageprocessing algorithms for extracting panoramic images and devicesimplementing the algorithms. In many cases, however, distortion-freerectilinear images can be more valuable. Or, it can be more satisfactorywhen panoramic images and rectilinear images are both available. FIG. 19is a conceptual drawing illustrating the rectilinear projection schemeof a prior art described in reference 9. A lens with a rectilinearprojection scheme is a so-called distortion-free lens, and thecharacteristics of a rectilinear lens are considered identical to thoseof a pinhole camera. To acquire an image with such a rectilinearprojection scheme, we assume an object plane(1931) and a processed imageplane(1935) in the world coordinate system as schematically shown inFIG. 19.

The imaging system in this embodiment is heading in an arbitrarydirection, and the third rectangular coordinate system takes the opticalaxis(1901) of the imaging system as the negative z″-axis, and the nodalpoint of the lens as the origin. Image sensor plane has a rectangularshape with a lateral width B and a longitudinal height V, and the imagesensor plane is a plane perpendicular to the optical axis. On the otherhand, the processed image plane has a rectangular shape with a lateralwidth W and a longitudinal height H. The x-axis of the first rectangularcoordinate system, the x′-axis of the second rectangular coordinatesystem, the x″-axis of the third rectangular coordinate system and theX-axis of the world coordinate system are all parallel to the sides ofthe image sensor plane along the lateral direction. Furthermore, thez-axis of the first rectangular coordinate system, the z′-axis of thesecond rectangular coordinate system, and the z″-axis of the thirdrectangular coordinate systems are all identical to each other and areheading to the exact opposite direction to the Z-axis of the worldcoordinate system.

In this embodiment, the processed image plane is assumed to be locatedat a distance s″ from the nodal point of the lens. In a rectilinearprojection scheme, the shape of the object plane(1931) is also a planeperpendicular to the optical axis, and the image of objects on theobject plane is faithfully reproduced on the processed image plane(1935)with both the lateral and the longitudinal scales preserved. The idealprojection scheme of a rectilinear lens is identical to the projectionscheme of a pinhole camera. Considering the simple geometricalcharacteristics of a pinhole camera, it is convenient to assume that theshape and the size of the object plane(1931) are identical to those ofthe processed image plane. Therefore, the distance from the objectplane(1931) to the nodal point N of the lens is also assumed as s″.

FIG. 20 illustrates the case where the intersection point ◯ between theimage sensor plane and the optical axis, or equivalently the thirdintersection point ◯″ on the processed image plane corresponding to thefirst intersection point ◯ does not coincide with the center C″ of theprocessed image plane(2043). Therefore, it corresponds to an imagingsystem with a slide operation as has been described in an embodiment ofthe prior art. In two-dimensional rectangular coordinate system havingthe third intersection point as the origin, the coordinate of the saidcenter C″ is given as (x″_(c), y″_(c)). Since the lateral dimension ofthe processed image plane is W, the lateral coordinate with respect tothe center C″ has a minimum value x″₁=−W/2 and a maximum value x″₂=W/2.Considering the coordinate of the center C″ on top of this, the range ofthe lateral coordinate of the processed image plane has a minimum valuex″₁=x″_(c)−W/2 and a maximum value x″₂=x″_(c)+W/2. Likewise, the rangeof the longitudinal coordinate has a minimum value y″₁=y″_(c)−H/2 and amaximum value y″₂=y″_(c)+H/2.

The distance between the third intersection point ◯″ on the processedimage plane to the third point P″, in other words, the image height r″is given by Eq. 37.r″=√{square root over ((x″)²+(y″)²)}{square root over((x″)²+(y″)²)}  [Equation 37]

Since the virtual distance from the nodal point of the lens to theprocessed image plane is s″, an incident ray arriving at the third pointby the rectilinear lens has a zenith angle given by Eq. 38.

$\begin{matrix}{\theta = {\tan^{- 1}\left( \frac{r^{''}}{s^{''}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 38} \right\rbrack\end{matrix}$

On the other hand, the azimuth angle of the said incident ray is givenby Eq. 39.

$\begin{matrix}{\phi = {\phi^{''} = {\tan^{- 1}\left( \frac{y^{''}}{x^{''}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 39} \right\rbrack\end{matrix}$

Therefore, when an incident ray having the said zenith angle and theazimuth angle forms an image point on the image sensor plane by theimage forming properties of the lens, the coordinate of the image pointis given by Eq. 40 and Eq. 41.x′=gr(θ)cos φ  [Equation 40]y′=gr(θ)sin φ  [Equation 41]

Therefore, it is only necessary to substitute the signal value of thethird point on the processed image plane by the signal value of theimage point on the uncorrected image plane having such rectangularcoordinate.

Similar to the embodiment of a prior art previously described,considering the facts that all the image sensors and the display devicesare digital devices, it is convenient to use the following set ofequations in image processing procedure. First of all, the size(I_(max), J_(max)) of the processed image plane and the horizontal FOVΔψ prior to any slide operation are determined. Then, the pixel distances″ between the nodal point of the lens and the processed image plane canbe obtained using Eq. 42.

$\begin{matrix}{s^{''} = \frac{J_{\max} - 1}{2\;{\tan\left( \frac{\Delta\psi}{2} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 42} \right\rbrack\end{matrix}$

Furthermore, the center coordinate of the processed image plane is givenby Eq. 43.

$\begin{matrix}{\left( {I_{o},J_{o}} \right) = \left( {\frac{1 + I_{\max}}{2},\frac{1 + J_{\max}}{2}} \right)} & \left\lbrack {{Equation}\mspace{14mu} 43} \right\rbrack\end{matrix}$

Here, Eq. 43 reflects the convention that the coordinate of the pixel onthe upper left corner of a digital image is designated as (1, 1).

Next, according to the needs, the displacement (ΔI, ΔJ) of the saidcenter from the third intersection point is determined. Once suchpreparatory stages have been finished, the zenith angle given in Eq. 44and the azimuth angle given in Eq. 45 are calculated for every pixel onthe processed image plane.

$\begin{matrix}{\theta_{I,J} = {\tan^{- 1}\left\{ \frac{\sqrt{\left( {I - I_{o} + {\Delta\; I}} \right)^{2} + \left( {J - J_{o} + {\Delta\; J}} \right)^{2}}}{s^{''}} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 44} \right\rbrack \\{\phi_{I,J} = {\tan^{- 1}\left( \frac{I - I_{o} + {\Delta\; I}}{J - J_{o} + {\Delta\; J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 45} \right\rbrack\end{matrix}$

Next, the image height r_(I,J) on the image sensor plane is calculatedusing Eq. 46.r _(I,J)=(θ_(I,J))  [Equation 46]

Next, the position of the second point on the uncorrected image plane iscalculated using the position (K_(o), L_(o)) of the second intersectionpoint on the uncorrected image plane and the magnification ratio g.x′ _(I,J) =L _(o) +gr _(I,J) cos(φ_(I,J))  [Equation 47]y′ _(I,J) =K _(o) +gr _(I,J) sin(φ_(I,J))  [Equation 48]

Once the position of the corresponding second point has been found, therectilinear image can be obtained using the previously describedinterpolation methods.

FIG. 21 is a rectilinear image extracted from the fisheye image given inFIG. 5, of which the lateral and the longitudinal dimensions are both250 pixels, and there is no slide operation. As can be seen from FIG.21, all the straight lines are captured as straight lines. On the otherhand, FIG. 22 is a rectilinear image extracted from FIG. 17 with thewidth:height ratio of 4:3. The position of the third intersection pointcoincides with the center of the processed image plane, and thehorizontal FOV is 60°. Here, it can be seen that all the straight linesin the world coordinate system are captured as straight lines in theprocessed image plane.

Panoramic imaging system in reference 8 requires a direction sensingmeans in order to provide natural-looking panoramic images at all thetimes irrespective of the inclination of the device having the imagingsystem with respect to the ground plane. However, it may happen thatadditional installation of a direction sensing means may be difficult interms of cost, weight, or volume for some devices such as motorcycle orunmanned aerial vehicle. FIG. 23 is a conceptual drawing illustratingthe definition of a multiple viewpoint panoramic image that can beadvantageously used in such cases.

An imaging system providing multiple viewpoint panoramic images iscomprised of an image acquisition means for acquiring an uncorrectedimage plane which is equipped with a wide-angle lens rotationallysymmetric about an optical axis, an image processing means for producinga processed image plane from the uncorrected image plane, and an imagedisplay means for displaying the processed image plane on a screen witha rectangular shape.

The processed image plane in FIG. 23 is composed of three subrectilinear image planes, namely, the 1^(st) sub rectilinear imageplane(2331-1), the 2^(nd) sub rectilinear image plane(2331-2) and the3^(rd) sub rectilinear image plane(2331-3). The 1^(st) through the3^(rd) sub rectilinear image planes are laid out horizontally on theprocessed image plane. More generally, the said processed image plane isa multiple viewpoint panoramic image, wherein the said multipleviewpoint panoramic image is comprised of the 1^(st) through the n^(th)sub rectilinear image planes laid out horizontally on the said screen, nis a natural number larger than 2, an arbitrary straight line in theworld coordinate system having the nodal point of the wide-angle lens asthe origin appears as a straight line(2381A) on any of the 1^(st)through the n^(th) sub rectilinear image plane, and any straight line inthe world coordinate system appearing on more than two adjacent subrectilinear image planes appears as a connected line segments(2381B-1,2381B-2, 2381B-3).

FIG. 24 is a conceptual drawing of an object plane providing a multipleviewpoint panoramic image. Object plane of the current embodiment has astructure where more than two planar sub object planes are joinedtogether. Although FIG. 24 is illustrated as a case where three subobject planes, namely 2431-1, 2431-2 and 2431-3 are used, a more generalcase of using n sub object planes can be easily understood as well. Inorder to easily understand the current embodiment, a sphere with aradius T centered at the nodal point N of the lens is assumed. If afolding screen is set-up around the sphere while keeping the foldingscreen to touch the sphere, then this folding screen corresponds to theobject plane of the current embodiment. Therefore, the n sub objectplanes are all at the same distance T from the nodal point of the lens.As a consequence, all the sub object planes have an identical zoom ratioor a magnification ratio.

In FIG. 24 using three sub object planes, the principal direction ofvision(2401-1) of the 1^(st) sub object plane(2431-1) makes an angle ofψ₁₋₂ with the principal direction of vision(2401-2) of the 2^(nd) subobject plane(2431-2), and the principal direction of vision(2401-3) ofthe 3^(rd) sub object plane(2431-3) makes an angle of ψ₃₋₄ with theprincipal direction of vision(2401-2) of the 2^(nd) sub objectplane(2431-2). The range of the horizontal FOV of the 1^(st) sub objectplane is from a minimum value ψ₁ to a maximum value ψ₂, and the range ofthe horizontal FOV of the 2^(nd) sub object plane is from a minimumvalue ψ₂ to a maximum value ψ₃. By having the horizontal FOVs ofadjacent sub object planes be seamlessly continued, a natural lookingmultiple viewpoint panoramic image can be obtained. The 1^(st) subobject plane and the 3^(rd) object plane are obtained by panning the2^(nd) sub object plane by appropriate angles.

FIG. 25 is another example of a fisheye image, and it shows the effectof installing a fisheye lens with 190° FOV on the ceiling of aninterior. On the other hand, FIG. 26 is a multiple viewpoint panoramicimage extracted from FIG. 25. Each sub object plane has a horizontal FOVof 190°/3. From FIG. 26, it can be seen that such an imaging system willbe useful as an indoor security camera.

FIG. 27 is a schematic diagram of an imaging system embodying theconception of the present invention and the invention of prior arts, andit is comprised of an image acquisition means(2710), an image processingmeans(2716), and an image display means(2717). The image processingmeans(2716) of the present invention has an input frame buffer(2771)storing one frame of image acquired from the image acquisitionmeans(2710). The input frame buffer(2771) stores a digital imageacquired from the image acquisition means(2710) in the form of atwo-dimensional array. This image is the uncorrected image plane. On theother hand, the output frame buffer(2773) stores an output signal in theform of a two-dimensional array, which corresponds to a processed imageplane(2735) that can be displayed on the image display means(2717). Acentral processing unit(2775) further exists, which generates aprocessed image plane from the uncorrected image plane existing in theinput frame buffer and stores the processed image plane in the outputframe buffer. The mapping relation between the output frame buffer andthe input frame buffer is stored in a non-volatile memory(2779) such asa SDRAM in the form of a lookup table (LUT). In other words, using thealgorithms from the embodiments of the current invention, a long list ofpixel addresses for the input frame buffer corresponding to particularpixels in the output frame buffer is generated and stored. Centralprocessing unit(2775) refers to this list stored in the nonvolatilememory in order to process the image.

On the other hand, an image selection device(2777) receives signalscoming from various sensors and image selection means and sends them tothe central processing unit. Also, by recognizing the button pressed bythe user, the image selection device can dictate whether the originaldistorted fisheye image is displayed without any processing, or apanoramic image with a cylindrical or a Mercator projection scheme isdisplayed, or a rectilinear image is displayed. Said nonvolatile memorystores a number of list corresponding to the number of possible optionsa user can choose.

In these various cases, the wide-angle lens rotationally symmetric aboutan axis that is mounted on the said image acquisition means can be arefractive fisheye lens with a FOV larger than 180°, but sometimes acatadioptric fisheye lens may be needed. For the projection scheme,equidistance projection scheme can be used, but stereographic projectionscheme can be used, also. Although, fisheye lenses with stereographicprojection schemes are preferable in many aspects in general, fisheyelenses with stereographic projection schemes are much harder both indesign and in manufacturing. Therefore, fisheye lenses with equidistanceprojection schemes can be realistic alternatives.

Depending on the application areas, the image display means(2717) can bea computer screen, a CCTV monitor, a CRT monitor, a digital television,an LCD projector, a network monitor, the display screen of a cellularphone, a navigation module for an automobile, and other various devices.

Such imaging systems have two drawbacks. Firstly, since image processingmeans needs additional components such as DSP chip or SDRAM, themanufacturing cost of the imaging system is increased. Secondly, imageprocessing takes more than several tens of milliseconds, and a time gapexist between the image displayed on the image display means and thecurrent state of the actual objects. Several tens of milliseconds is nota long time, but it can correspond to several meters for an automobiledriving at a high speed. Therefore, application areas exist where evensuch a short time gap is not allowed. The gap will be greater in otherapplication areas such as airplanes and missiles.

Reference 10 discloses a technical conception for providing panoramicimages without image processing by deliberately matching the pixellocations within an image sensor to desired panoramic images. Shown inFIG. 28 is a schematic diagram of a general catadioptric panoramicimaging system. As schematically shown in FIG. 28, a catadioptricpanoramic imaging system of prior arts includes as constituent elementsa rotationally symmetric panoramic mirror(2811), of which thecross-sectional profile is close to an hyperbola, a lens(2812) that islocated on the rotational-symmetry axis(2801) of the mirror(2811) andoriented toward the said mirror(2811), and a camera body(2814) having animage sensor(2813) inside. Then, an incident ray(2805) having a zenithangle θ, which originates from every 360° directions around the mirrorand propagates toward the rotational-symmetry axis(2801), is reflectedat a point M on the mirror surface(2811), and captured by the imagesensor(2813). The image height is given as r=r(θ).

FIG. 29 is a conceptual drawing of an exemplary rural landscapeobtainable using the catadioptric panoramic imaging system of prior artschematically shown in FIG. 28. As illustrated in FIG. 29, aphotographic film or an image sensor(2813) has a square or a rectangularshape, while a panoramic image obtained using a panoramic imaging systemhas an annular shape. Non-hatched region in FIG. 29 constitutes apanoramic image, and the hatched circle in the center corresponds to thearea at the backside of the camera, which is not captured because thecamera body occludes its view. An image of the camera itself reflectedby the mirror(2811) lies within this circle. On the other hand, thehatched regions at the four corners originate from the fact that thediagonal field of view of the camera lens(2812) is larger than the fieldof view of the panoramic mirror(2811). The image of the scene in frontof the camera that is observable in absence of the panoramic mirror liesin these regions. FIG. 30 is an exemplary unwrapped panoramic imageobtained from the ring-shaped panoramic image in FIG. 29 by cuttingalong the cutting-line and converting into a perspectively normal viewusing image processing software.

FIG. 31 shows geometrical relations necessary to transform thering-shaped panoramic image illustrated in FIG. 29 into a perspectivelynormal unwrapped panoramic image shown in FIG. 30. Origin O of therectangular coordinate system lies at the center of the image sensorplane(3113), the x-coordinate increases from the left side of the imagesensor plane to the right side of the image sensor plane, and they-coordinate increases from the top end of the image sensor plane to thebottom end of the image sensor plane. Image plane(3133) on the imagesensor plane(3113) has a ring shape defined by a concentric innerrim(3133 a) and an outer rim(3133 b). The radius of the inner rim(3133a) is r₁ and the radius of the outer rim(3133 b) is r₂. The centercoordinate of the panoramic image plane(3133) in a rectangularcoordinate system is (0, 0), and the coordinate of a point P on thepanoramic image plane defined by the inner rim(3133 a) and the outerrim(3133 b) is given as (x, y). On the other hand, the coordinate of thesaid point P in a polar coordinate is given as (r, θ). Variables in therectangular coordinate system and the polar coordinate system satisfysimple geometrical relations given in Eq. 49 through Eq. 52. Using theserelations, variables in the two coordinate systems can be readilytransformed into each other.

$\begin{matrix}{x = {r\;\cos\;\phi}} & \left\lbrack {{Equation}\mspace{14mu} 49} \right\rbrack \\{y = {r\;\sin\;\phi}} & \left\lbrack {{Equation}\mspace{14mu} 50} \right\rbrack \\{r = \sqrt{x^{2} + y^{2}}} & \left\lbrack {{Equation}\mspace{14mu} 51} \right\rbrack \\{\phi = {\tan^{- 1}\left( \frac{y}{x} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 52} \right\rbrack\end{matrix}$

Here, the lateral coordinate on the unwrapped panoramic image isproportional to the azimuth angle φ, and the longitudinal coordinate isproportional to radius r.

As described previously, reference 10 discloses a CMOS image sensorwhich does not require image processing in order to transformring-shaped panoramic image into a rectangular unwrapped panoramicimage.

As schematically shown in FIG. 32, ordinary CMOS image sensorplane(3213) is comprised of multitude of pixels(3281) arranged in amatrix form, a vertical shift register(3285), and a horizontal shiftregister(3284). Each pixel(3281) is comprised of a photodiode(3282)converting the received light into electrical charges proportional toits light intensity, and a photoelectrical amplifier(3283) convertingthe electrical charges (i.e., the number of electrons) into anelectrical voltage. The light captured by a pixel is converted into anelectrical voltage at the pixel level, and outputted into a signal lineby the vertical and the horizontal shift registers. Here, the verticalshift register can selectively choose among the pixels belonging todifferent rows, and the horizontal shift register can selectively chooseamong the pixels belonging to different columns. A CMOS image sensorhaving such a structure operates in a single low voltage, and consumesless electrical power.

A photoelectrical amplifier(3283) existing in each pixel(3281) occupiesa certain area which does not contribute to light capturing. The ratioof the light receiving area (i.e., the photodiode) over the pixel areais called the Fill Factor, and if the Fill Factor is large, then thelight detection efficiency is high.

A CMOS image sensor having such structural characteristics has a higherlevel of freedom in design and pixel arrangement compared to a CCD imagesensor. FIG. 33 is a floor plan showing pixel arrangement in a CMOSimage sensor for panoramic imaging system according to an embodiment ofthe invention of a prior art. The CMOS image sensor has an imagingplane(3313) whereon pixels(3381) are arranged. Although pixels(3381) inFIG. 33 have been represented as dots, these dots are merely the centerpositions representing the positions of the pixels, and actualpixels(3381) occupy finite areas around the said dots. Each pixel(3381)is positioned on intersections between M radial lines(3305) having aposition O on the image sensor plane as a starting point and Nconcentric circles(3306) having the said one point ◯ as the commoncenter. M and N are both natural numbers, and the radius r_(n) of then^(th) concentric circle from the center position O is given by Eq. 53.

$\begin{matrix}{r_{n} = {{\frac{n - 1}{N - 1}\left( {r_{N} - r_{1}} \right)} + r_{1}}} & \left\lbrack {{Equation}\mspace{14mu} 53} \right\rbrack\end{matrix}$

In Eq. 53, n is a natural number ranging from 1 to N, r₁ is the radiusof the first concentric circle from the center position O, and r_(n) isthe radius of the n^(th) concentric circle from the center position O.All the M radial lines maintain an identical angular distance. In otherwords, pixels on a given concentric circle are positioned along theperimeter of the concentric circle maintaining an identical interval.Furthermore, by Eq. 53, each pixel is positioned along a radial linemaintaining an identical interval.

Considering the characteristics of panoramic images, it is better if theimage sensor plane(3313) has a square shape. Preferably, each side of animage sensor plane(3313) is not smaller than 2r_(N), and the center O ofthe concentric circles is located at the center of the image sensorplane(3313).

A CMOS image sensor for panoramic imaging system illustrated in FIG. 33have pixels(3381) arranged at regular interval along the perimeters ofconcentric circles(3306), wherein the concentric circles have a regularinterval along the radial direction, in turn. Although not schematicallyillustrated in FIG. 33, such a CMOS image sensor further has a radialshift register or r-register and a circumferential shift register orθ-register. Pixels belonging to different concentric circles can beselected using the radial shift register, and pixels belonging todifferent radial lines on a concentric circle can be selected using thecircumferential shift register.

Every pixel on the previously described CMOS image sensor according toan embodiment of the invention of a prior art has a photodiode with anidentical size. On the other hand, pixel's illuminance decreases fromthe center of the CMOS image sensor plane(3313) toward the boundaryaccording to the well-known cosine fourth law. Cosine fourth law becomesan important factor when an exposure time has to be determined for acamera lens with a wide field-of-view, or the image brightness ratiobetween the pixel located at the center of the image sensor plane andthe pixel at the boundary has to be determined. FIG. 34 shows thestructure of a CMOS image sensor for panoramic imaging system forresolving the illuminance difference between the center and the boundaryof an image sensor plane according to another embodiment of theinvention of a prior art. Identical to the embodiment previouslydescribed, each pixel is positioned at a regular interval on radiallines, wherein the radial lines in turn have an identical angulardistance on a CMOS image sensor plane(3413). In other words, each pixelis positioned along the perimeter of a concentric circle at a regularinterval, and the concentric circles on the image sensor plane have anidentical interval along the radial lines. Although pixels in FIG. 34have been represented as dots, these dots merely represent the centerpositions of pixels, and actual pixels occupy finite areas around thesedots.

Since all the pixels on the image sensor plane have an identical angulardistance along the perimeter, the maximum area on the image sensor planewhich a particular pixel can occupy increases proportionally to theradius. In other words, a pixel(3430 j) existing on a circle on theimage sensor plane with a radius r_(j) can occupy a larger area than apixel(3430 i) existing on a circle with a radius r_(i) (wherein,r_(i)<r_(j)). Since pixels located far from the center of the imagesensor plane can occupy large areas than pixels located near the center,if each pixel's photoelectrical amplifier is identical in size, thenpixels near the boundary of the image sensor plane far from the centercan have relatively larger photodiodes. By having a larger lightreceiving area, the fill factor can be increased, and the decrease inilluminance by the cosine fourth law can be compensated. In this case,the size of the photodiode can be proportional to the radius of theconcentric circle.

On the other hand, reference 11 discloses a structure of a CMOS imagesensor for compensating the lens brightness decreasing as it moves awayfrom the optical axis by making the photodiode area becomes larger as itmoves away from the optical axis.

-   [reference 1] J. F. Blinn and M. E. Newell, “Texture and reflection    in computer generated images”, Communications of the ACM, 19,    542-547 (1976).-   [reference 2] N. Greene, “Environment mapping and other applications    of world projections”, IEEE Computer Graphics and Applications, 6,    21-29 (1986).-   [reference 3] E. W. Weisstein, “Cylindrical Projection”,    http://mathworld.wolfram.com/CylindricalProjection.html.-   [reference 4] W. D. G. Cox, “An introduction to the theory of    perspective—part 1”, The British Journal of Photography, 4, 628-634    (1969).-   [reference 5] G. Kweon and M. Laikin, “Fisheye lens”, Korean patent    10-0888922, date of patent Mar. 10, 2009.-   [reference 6] G. Kweon, Y. Choi, and M. Laikin, “Fisheye lens for    image processing applications”, J. of the Optical Society of Korea,    12, 79-87 (2008).-   [reference 7] G. Kweon and M. Laikin, “Wide-angle lenses”, Korean    patent 10-0826571, date of patent Apr. 24, 2008.-   [reference 8] G. Kweon, “Methods of obtaining panoramic images using    rotationally symmetric wide-angle lenses and devices thereof”,    Korean patent 10-0882011, date of patent Jan. 29, 2009.-   [reference 9] G. Kweon, “Method and apparatus for obtaining    panoramic and rectilinear images using rotationally symmetric    wide-angle lens”, Korean patent 10-0898824, date of patent May 14,    2009.-   [reference 10] G. Kweon, “CMOS image sensor and panoramic imaging    system having the same”, Korean patent 10-0624051, date of patent    Sep. 7, 2006.-   [reference 11] A. Silverstein, “Method, apparatus, and system    providing a rectilinear pixel grid with radially scaled pixels”,    international application number PCT/US2008/060185, date of    international application Apr. 14, 2008.

DISCLOSURE Technical Problem

The purpose of the present invention is to provide image processingalgorithms and imaging systems for extracting industrially usefulcomplex images from digitized images acquired using a camera equippedwith a wide-angle lens that is rotationally symmetric about an opticalaxis and hard-wired CMOS image sensors that can be used in imagingsystems which do not require software image processing in order toprovide such images.

Technical Solution

The present invention provides image processing algorithms that areaccurate in principle based on the geometrical optics principleregarding image formation by wide-angle lenses with distortion andmathematical definitions of industrially useful images, and thisinvention also provide CMOS image sensors which physically implementsuch algorithms.

Advantageous Effects

Since image processing is executed within the CMOS image sensor by meansof the physical structure, there is no delay of video signal, andpanoramic or wide-angle imaging systems can be realized withconsiderably lower costs compared to imaging systems relying on softwareimage processing.

DESCRIPTION OF DRAWINGS

FIG. 1 is a conceptual drawing of the latitude and the longitude.

FIG. 2 is a conceptual drawing of a map with an equi-rectangularprojection scheme.

FIG. 3 is a conceptual drawing illustrating a cylindrical projectionscheme.

FIG. 4 is a conceptual drawing illustrating the real projection schemeof a general rotationally symmetric lens.

FIG. 5 is an exemplary image produced by a computer assuming that afisheye lens with an equidistance projection scheme has been used totake the picture of an imaginary scene.

FIG. 6 is a diagram showing the optical structure of a refractivefisheye lens with a stereographic projection scheme along with thetraces of rays.

FIG. 7 is a diagram showing the optical structure of a catadioptricfisheye lens with a stereographic projection scheme along with thetraces of rays.

FIG. 8 is a diagram showing the optical structure of a catadioptricpanoramic lens with a rectilinear projection scheme along with thetraces of rays.

FIG. 9 is a conceptual drawing of the world coordinate system of theinvention of a prior art.

FIG. 10 is a schematic diagram of an imaging system of the invention ofa prior art relying on software image processing.

FIG. 11 is a conceptual drawing of an uncorrected image plane.

FIG. 12 is a conceptual drawing of a processed image plane that can beshown on an image display means.

FIG. 13 is a conceptual drawing of an object plane assumed by apanoramic imaging system having a cylindrical projection schemeaccording to an embodiment of the invention of a prior art.

FIG. 14 is a cross-sectional diagram of the object plane shown in FIG.13 in X-Z plane.

FIG. 15 is a cross-sectional diagram of the object plane shown in FIG.13 in Y-Z plane.

FIG. 16 is an exemplary panoramic image following a cylindricalprojection scheme extracted from FIG. 5.

FIG. 17 is an exemplary image of an interior scene captured using afisheye lens from the invention of a prior art.

FIG. 18 is a panoramic image with a horizontal FOV of 190° and followinga cylindrical projection scheme extracted from the fisheye image givenin FIG. 17.

FIG. 19 is a conceptual drawing illustrating a rectilinear projectionscheme according to the invention of a prior art.

FIG. 20 is a conceptual drawing illustrating the change in field of viewas the relative position of the processed image plane is changed.

FIG. 21 is an exemplary rectilinear image with a horizontal FOV of 120°extracted from FIG. 5.

FIG. 22 is a rectilinear image with a horizontal FOV of 60° extractedfrom the fisheye image given in FIG. 17.

FIG. 23 is a conceptual drawing illustrating the definition of amultiple viewpoint panoramic image.

FIG. 24 is a conceptual drawing of an object plane corresponding to amultiple viewpoint panoramic image.

FIG. 25 is another exemplary image of an interior scene captured using afisheye lens from the invention of a prior art.

FIG. 26 is an exemplary multiple viewpoint panoramic image extractedfrom the fisheye image given in FIG. 25.

FIG. 27 is a schematic diagram of a preferable embodiment of an imageprocessing means according to the invention of a prior art.

FIG. 28 is a schematic diagram of a catadioptric panoramic imagingsystem in prior arts.

FIG. 29 is a conceptual drawing of an exemplary raw panoramic imageacquired using the catadioptric panoramic imaging system schematicallyshown in FIG. 28.

FIG. 30 is an unwrapped panoramic image corresponding to the rawpanoramic image given in FIG. 29.

FIG. 31 is a schematic diagram for understanding the geometricaltransformation of panoramic images.

FIG. 32 is a schematic diagram illustrating the structure of an ordinaryCMOS image sensor.

FIG. 33 is a conceptual diagram illustrating pixel arrangement in a CMOSimage sensor for panoramic imaging system according to an embodiment ofthe invention of a prior art.

FIG. 34 is a conceptual diagram illustrating the structure of a CMOSimage sensor for panoramic imaging system according to an embodiment ofthe invention of a prior art.

FIG. 35 is a conceptual diagram of an object plane according to thefirst embodiment of the current invention.

FIG. 36 is a conceptual diagram of an object plane corresponding to acylindrical panoramic image plane according to the first embodiment ofthe current invention.

FIG. 37 is a conceptual diagram of an object plane corresponding to amultiple viewpoint panoramic image plane according to the firstembodiment of the current invention.

FIG. 38 is an example of a multiple viewpoint panoramic image planecorresponding to FIG. 37.

FIG. 39 is a conceptual diagram of an object plane corresponding to acomplex image plane according to the first embodiment of the currentinvention.

FIG. 40 is an example of a complex image plane extracted from FIG. 5corresponding to the object plane in FIG. 39.

FIG. 41 is another example of a complex image plane extracted from FIG.17 corresponding to the object plane in FIG. 39.

FIG. 42 is a cross-sectional diagram of a complex object plane in theY-Z plane according to the second embodiment of the current invention.

FIG. 43 is an example of a complex image plane extracted from FIG. 5according to the second embodiment of the current invention.

FIG. 44 is an image of an exterior scene captured using a fisheye lensfrom the invention of a prior art.

FIG. 45 is an example of a complex image plane extracted from FIG. 44according to the second embodiment of the current invention.

FIG. 46 is a conceptual diagram of a complex image plane according tothe third embodiment of the current invention.

FIG. 47 is a conceptual diagram of a complex object plane correspondingto the complex image plane in FIG. 46.

FIG. 48 is an example of a complex image plane extracted from thefisheye image in FIG. 44.

FIG. 49 is a conceptual diagram showing the dependence of the objectplane in FIG. 47 on the Z-axis.

FIG. 50 is an example of a fisheye image following a stereographicprojection scheme which has been extracted from FIG. 51 according to thefifth embodiment of the current invention.

FIG. 51 is an exemplary image of an interior scene captured using afisheye lens of the invention of a prior art.

FIG. 52 is an example of a fisheye image following a stereographicprojection scheme which has been extracted from FIG. 51 according to thefifth embodiment of the current invention.

FIG. 53 is an example of a fisheye image following an equidistanceprojection scheme which has been extracted from FIG. 5 according to thefifth embodiment of the current invention.

FIG. 54 is another example of a complex image which has been extractedfrom FIG. 5 according to the sixth embodiment of the current invention.

FIG. 55 is an exemplary disposition of pixel centers on a CMOS imagesensor plane according to the seventh embodiment of the currentinvention.

FIG. 56 is a drawing showing the pixels in FIG. 55 only.

FIG. 57 is a diagram for understanding the pixel distribution on theimage sensor plane in a CMOS image sensor according to the seventhembodiment of the current invention.

FIG. 58 is a conceptual diagram for understanding the direction of anopen curved line in the current invention.

FIG. 59 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following an equi-rectangular projection schemeaccording to the seventh embodiment of the current invention.

FIG. 60 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following a Mercator projection scheme according tothe seventh embodiment of the current invention.

FIG. 61 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following a cylindrical projection scheme accordingto the seventh embodiment of the current invention, where the realdistortion characteristics of a lens are reflected in the pixeldistribution.

FIG. 62 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following a rectilinear projection scheme accordingto the eight embodiment of the current invention.

FIG. 63 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following a rectilinear projection scheme accordingto the eight embodiment of the current invention.

FIG. 64 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following a multiple viewpoint panoramic projectionscheme according to the ninth embodiment of the current invention.

FIG. 65 is an example of pixel distribution on an image sensor plane ina CMOS image sensor following a complex projection scheme according tothe ninth embodiment of the current invention.

FIG. 66 is a conceptual diagram illustrating the pixel structure in anordinary CMOS image sensor.

FIG. 67 is a schematic diagram illustrating a case of forminginsensitive pixels in FIG. 64 in order to mark boundary lines betweensub image planes.

FIG. 68 is a schematic diagram illustrating a case of forminginsensitive pixels in FIG. 65 in order to mark boundary lines betweensub image planes.

MODE FOR INVENTION

Referring to FIG. 35 through FIG. 68, the preferable embodiments of thepresent invention will be described in detail.

First Embodiment

FIG. 35 is a conceptual diagram of an object plane(3531) having a shapeof a generalized cylinder according to the first embodiment of thecurrent invention. Such an imaginary object plane can be easilyunderstood by assuming that it can be easily bended or folded but itcannot be stretched or shrunken. On the other hand, if this object planeis not bended or folded, then the shape of the object plane is identicalto that of the processed image plane. In other words, when an objectplane is flattened out, this object plane becomes identical to aprocessed image plane.

As has been previously described, an object plane exists in the worldcoordinate system in an arbitrarily bended or folded state. The threecoordinates in the world coordinate system describing an imaginaryobject point Q on an object plane are given as X, Y, Z, and there areminor differences with the world coordinate system described inreferences 8 or 9. In other words, the origin of the world coordinatesystem of the current invention lies at the nodal point N of awide-angle lens, the Z-axis coincides with the optical axis(3501) of thewide-angle lens, and the Y-axis passes through the said origin and isparallel to the sides of the image sensor plane within the camera bodyalong the longitudinal direction, where the direction from the top endto the bottom end of the image sensor plane is the positive(+)direction. Or, when the object plane is seen from the origin, thepositive direction is from top to bottom. On the other hands, the X-axisis parallel to the sides of the image sensor plane(3513) along thelateral direction, and when the object plane(3531) is seen from theorigin N, the positive direction is from left to right. The said worldcoordinate system is a right-handed coordinate system. Therefore, if thedirections of the Z-axis and the Y-axis are determined, then thedirection of the X-axis is automatically determined.

On the other hands, the reference point O in the first rectangularcoordinate system describing image points on the image sensor plane liesat the intersection point between the optical axis(3501) and the imagesensor plane(3513). The x-axis is parallel to the sides of the imagesensor plane along the lateral direction, and when the image sensorplane is seen from the nodal point of the wide-angle lens, thepositive(+) direction runs from the left side of the image sensor planeto the right side. The y-axis is parallel to the sides of the imagesensor plane along the longitudinal direction, and when the image sensorplane is seen from the nodal point of the wide-angle lens, thepositive(+) direction runs from the top end of the image sensor plane tothe bottom end. On the other hand, the direction of the x′-axis in thesecond rectangular coordinate system describing the uncorrected imageplane and the direction of the x″-axis in the third rectangularcoordinate system describing the processed image plane are the same asthe direction of the x-axis in the first rectangular coordinate system.Furthermore, the direction of the y′-axis in the second rectangularcoordinate system and the direction of the y″-axis in the thirdrectangular coordinate system are the same as the direction of they-axis in the first rectangular coordinate system.

Since flattened-out object plane is identical to the processed imageplane, each pixel (I, J) on the processed image plane has acorresponding imaginary object point on the imaginary object plane.Therefore, it is possible to allocate a two-dimensional coordinate (I,J) on the object plane. An object point Q=Q(I, J) having atwo-dimensional coordinate (I, J) on the object plane has acorresponding image point P=P(I, J) on the processed image plane. On theother hand, an object point Q=Q(I, J) having a coordinate (I, J) on theobject plane has a three-dimensional space coordinate X_(I,J), Y_(I,J),Z_(I,J)) in the world coordinate system. Each coordinate X_(I,J),Y_(I,J), Z_(I,J) is a function of both I and J. In other words,X_(I,J)=X(I, J), Y_(I,J)=Y(I, J), and Z_(I,J)=Z(I, J).

If the lateral pixel coordinate on the object plane changes by 1 from(I, J) to (I, J+1), then the three-dimensional space coordinates alsochange from (X_(I,J), Y_(I,J), Z_(I,J)) to (X_(I,J+1), Y_(I,J+1),Z_(I,J+1)). As has been stated, the object plane is assumed neitherstretchable nor shrinkable. Therefore, provided the object plane is notfolded and the degree of bending is relatively minute, if thetwo-dimensional pixel coordinates in the object plane changes by 1, thenthe three-dimensional space coordinates also change about by 1 indistance. More precisely, the space coordinate in the world coordinatesystem can change about by a constant C. Since it does not affect theimage processing method, however, it is assumed that the position ischanged by 1 in distance for simplicity of argument. Therefore, therelation in Eq. 54 holds true.(X _(I,J+1) −X _(I,J))²+(Y _(I,J+1) −Y _(I,J))²+(X _(I,J+1) −X_(I,J))²≅1  [Equation 54]

Similarly, if the longitudinal pixel coordinate changes by 1 from (I, J)to (I+1, J), then the three-dimensional space coordinates in the worldcoordinate system change about by 1 in distance. Therefore, the relationin Eq. 55 holds true.(X _(I+1,J) −X _(I,J))²+(Y _(I+1,J) −Y _(I,J))²+(X _(I+1,J) −X_(I,J))²≅1  [Equation 55]

The pixel coordinate on the processed image plane and on the objectplane is given as

(I, J), where I and J are natural numbers. Mathematically, however, itcan be assumed that the pixel coordinate can take real numbers. The sizeof the processed image plane is given as a two-dimensional matrix havingI_(max) rows and J_(max) columns. Each pixel is a square pixel having ashape of a right-rectangle. If we assume the pixel coordinate given asnatural numbers are the center position of a square pixel, then therange of the longitudinal coordinate I of the two-dimensional coordinateon the object plane is given as (0.5≦I≦I_(max)+0.5), and the range ofthe lateral coordinate J is given as (0.5≦J≦J_(max)+0.5). Using the factthat the object plane is neither stretchable nor shrinkable, and theassumption that the two-dimensional coordinate I and J are real-numberedvariables, Eq. 54 is reduced to Eq. 56, and Eq. 55 is reduced to Eq. 57.

$\begin{matrix}{{\left( \frac{\partial X_{I,J}}{\partial J} \right)^{2} + \left( \frac{\partial Y_{I,J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial J} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 56} \right\rbrack \\{{\left( \frac{\partial X_{I,J}}{\partial I} \right)^{2} + \left( \frac{\partial Y_{I,J}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial I} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 57} \right\rbrack\end{matrix}$

Such a shape of the object plane automatically determines the projectionscheme of the imaging system. The coordinate of an object point Q=Q(X,Y, Z) in the three-dimensional space is given as Q=Q(R, θ, φ) in aspherical polar coordinate system. The rectangular coordinate given as(X, Y, Z) and the spherical polar coordinate given as (R, θ, φ) satisfythe following equations.X=R sin θ cos φ  [Equation 58]Y=R sin θ sin φ  [Equation 59]Z=R cos θ  [Equation 60]R=√{square root over (X ² +Y ² +Z ²)}  [Equation 61]

Therefore, from the three-dimensional rectangular coordinate, theazimuth angle φ of an incident ray originating from the said objectpoint Q is given by Eq. 62 and the zenith angle θ is given by Eq. 63.

$\begin{matrix}{\phi = {\tan^{- 1}\left( \frac{Y}{X} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 62} \right\rbrack \\{\theta = {\cos^{- 1}\left( \frac{Z}{\sqrt{X^{2} + Y^{2} + Z^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 63} \right\rbrack\end{matrix}$

Therefore, it is only necessary to substitute the signal value of animaginary image point on the uncorrected image plane having thecoordinates given by Eq. 64 and Eq. 65 for the signal value of an imagepoint having a pixel coordinate (I, J) on the processed image plane.x′=gr(θ)cos φ  [Equation 64]y′=gr(θ)sin φ  [Equation 65]

Using Eq. 62 through Eq. 65, a complex image having an ideal projectionscheme can be obtained from a fisheye image having a distortion. First,according to the user's need, the size (W, H) and the curved or bendedshape of a desirable object plane, and its location within the worldcoordinate system are determined. As has been stated previously,determining the shape and the location of an object plane is equivalentto designing a projection scheme of a complex image. Then, the azimuthangle φ and the zenith angle θ of an incident ray corresponding to athree-dimensional rectangular coordinate (X(I, J), Y(I, J), Z(I, J)) onthe object plane having a two-dimensional coordinate (I, J) can beobtained using Eq. 62 and Eq. 63. Next, an image height r correspondingto the zenith angle θ of this incident ray is obtained. The rectangularcoordinate (x′, y′) of the image point on the uncorrected image planecorresponding to this image height r, the magnification ratio g, and theazimuth angle φ of the incident ray is obtained using Eq. 64 and Eq. 65.Finally, the video signal (i.e., RGB signal) value of the image point bythe fisheye lens having this rectangular coordinate is substituted forthe video signal value of the image point on the said complex imagehaving a pixel coordinate (I, J). A complex image having a designedprojection scheme can be obtained by image processing for all the pixelsin the complex image by this method.

Considering the fact that an image sensor plane, an uncorrected imageplane and a processed image plane are all digitized, a complex imageacquisition device of the first embodiment of the current invention iscomprised of an image acquisition means for acquiring an uncorrectedimage plane using a camera equipped with a wide-angle lens rotationallysymmetric about an optical axis, an image processing means forgenerating a processed image plane based on the said uncorrected imageplane, and an image display means for displaying the said processedimage plane. The said uncorrected image plane is a two-dimension arraywith K_(max) rows and L_(max) columns, the pixel coordinate of theoptical axis on the said uncorrected image plane is (K_(o), L_(o)), andthe real projection scheme of the said wide-angle lens is given by afunction such as given in Eq. 66.r=r(θ)  [Equation 66]

Here, the real projection scheme of a lens is the image height robtained as a function of the zenith angle θ of the correspondingincident ray. The magnification ratio g of the said camera is given byEq. 67, where r′ is a pixel distance on the uncorrected image planecorresponding to the image height r.

$\begin{matrix}{g = \frac{r^{\prime}}{r}} & \left\lbrack {{Equation}\mspace{14mu} 67} \right\rbrack\end{matrix}$

The said processed image plane and the object plane are two-dimensionalarrays with I_(max) rows and J_(max) columns. The shape of an objectplane corresponding to a desirable complex image is determined using Eq.68 through Eq. 70.X _(I,J) =X(I,J)  [Equation 68]Y _(I,J) =Y(I,J)  [Equation 69]Z _(I,J) =Z(I,J)  [Equation 70]

The rectangular coordinates given by Eq. 68 through Eq. 70 must satisfyEq. 71 and Eq. 72.

$\begin{matrix}{{\left( \frac{\partial X_{I,J}}{\partial J} \right)^{2} + \left( \frac{\partial Y_{I,J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial J} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 71} \right\rbrack \\{{\left( \frac{\partial X_{I,J}}{\partial I} \right)^{2} + \left( \frac{\partial Y_{I,J}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial I} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 72} \right\rbrack\end{matrix}$

However, in a case where the object plane is folded, derivatives do notexist on folded points, and Eq. 71 or Eq. 72 cannot be applied at suchpoints. Therefore, the shape of such an object plane can be summarizedas follows. The shape of an object plane according to the firstembodiment of the present invention, i.e., X=X(I, J), Y=Y(I, J) andZ=Z(I, J), are continuous functions of I and J, and satisfy Eq. 71 andEq. 72 on differentiable points.

Once the shape of the object plane is determined, azimuth angles givenby Eq. 73 and zenith angles given by Eq. 74 are obtained for all objectpoints Q=Q(I, J) on the object plane having coordinates with naturalnumbers.

$\begin{matrix}{\phi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}}{X_{I,J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 73} \right\rbrack \\{\theta_{I,J} = {\cos^{- 1}\left( \frac{Z_{I,J}}{\sqrt{X_{I,J}^{2} + Y_{I,J}^{2} + Z_{I,J}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 74} \right\rbrack\end{matrix}$

The image height on the image sensor plane is obtained using Eq. 75.r _(I,J) =r(θ_(I,J))  [Equation 75]

Using the second intersection point on the uncorrected image plane, inother words, the pixel coordinate (K_(o), L_(o)) of the optical axis andthe magnification ratio g, the location of the second point on theuncorrected image plane, in other words, the location of the imaginaryimage point is determined.x′ _(I,J) =L _(o) +gr _(I,J) cos φ_(I,J)  [Equation 76]y′ _(I,J) =K _(o) +gr _(I,J) sin φ_(I,J)  [Equation 77]

Once the position of the corresponding second point is found, panoramicimages can be obtained using various interpolation methods such asnearest-neighbor, bilinear interpolation, or bicubic interpolationmethod.

The shape of the object plane for obtaining a rectilinear image such asthe one given in the first embodiment in reference 9 is given by Eq. 78through Eq. 80. Here, the size of the processed image plane is (I_(max),J_(max))=(250, 250), and (I_(o), J_(o)) is the center position of theprocessed image plane. The processed image plane is set-up so that it isparallel to the X-Y plane and perpendicular to the optical axis.Furthermore, the horizontal field of view of the processed image planeis designed as Δψ=120°, and for this purpose, the distance s″ from theorigin N of the world coordinate system to the object plane is given byEq. 80.

$\begin{matrix}{X_{I,J} = {J - J_{o}}} & \left\lbrack {{Equation}\mspace{14mu} 78} \right\rbrack \\{Y_{I,J} = {I - I_{o}}} & \left\lbrack {{Equation}\mspace{14mu} 79} \right\rbrack \\{Z_{I,J} = {s^{''} = \frac{J_{\max} - 1}{2\;{\tan\left( \frac{\Delta\psi}{2} \right)}}}} & \left\lbrack {{Equation}\mspace{14mu} 80} \right\rbrack\end{matrix}$

It can be easily confirmed that the shape of the object plane given byEq. 78 through Eq. 80 satisfy Eq. 71 and Eq. 72. Furthermore, arectilinear image extracted using an algorithm according to the firstembodiment of the present invention coincides with FIG. 21. Therefore,it can be seen that a rectilinear image is merely one example of thefirst embodiment of the present invention.

On the other hand, FIG. 36 is a cross-sectional diagram of an imaginaryobject plane in the X-Z plane corresponding to the cylindrical panoramicimage plane given in the first embodiment in reference 8, and forconvenience, the horizontal FOV ΔV is assumed as 180°. Since the lateralsize of the processed image plane and the object plane is given asJ_(max)−1, the axial radius ρ of the object plane is given by Eq. 81.

$\begin{matrix}{\rho = \frac{J_{\max} - 1}{\Delta\psi}} & \left\lbrack {{Equation}\mspace{14mu} 81} \right\rbrack\end{matrix}$

On the other hand, the center coordinate of the object plane is given by(I_(o), J_(o)), and the Z-axis passes through the center of the objectplane. Therefore, the horizontal field of view ψ_(I,J) an object pointhaving a lateral pixel coordinate J and a longitudinal pixel coordinateI makes with the Y-Z plane is given by Eq. 82.

$\begin{matrix}{\psi_{I,J} = \frac{J - J_{o}}{\rho}} & \left\lbrack {{Equation}\mspace{14mu} 82} \right\rbrack\end{matrix}$

Therefore, the coordinate X_(I,J) in the X-axis direction is given byEq. 83, and the coordinate Z_(I,J) in the Z-axis direction is given byEq. 84.

$\begin{matrix}{X_{I,J} = {\rho\;{\sin\left( \frac{J - J_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 83} \right\rbrack \\{Z_{I,J} = {\rho\;{\cos\left( \frac{J - J_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 84} \right\rbrack\end{matrix}$

On the other hand, since the shape of the object plane is a cylinder,the coordinate Y_(I,J) in the Y-axis direction is given by Eq. 85.Y _(I,J) =I−I _(o)  [Equation 85]

A processed image plane corresponding to such an object plane isidentical to FIG. 16. Therefore, a cylindrical panoramic image is alsoan example of the first embodiment of the present invention. An objectplane corresponding to a cylindrical panoramic image takes a shape thatis bended around the Z-axis with an axial radius ρ.

FIG. 37 depicts a multiple viewpoint panoramic image plane in FIG. 24.While the object plane in FIG. 36 is bended in curves, the objectplane(3731) in FIG. 37 is folded and is identical to the multipleviewpoint panoramic object plane in FIG. 24. The coordinates of thefolding points are given as in the following Eq. 86 through Eq. 90.

$\begin{matrix}{T = \frac{J_{\max} - 1}{6\mspace{14mu}\tan\frac{\pi}{6}}} & \left\lbrack {{Equation}\mspace{14mu} 86} \right\rbrack \\{X_{4} = {{- X_{1}} = \frac{T}{\cos\frac{\pi}{6}}}} & \left\lbrack {{Equation}\mspace{14mu} 87} \right\rbrack \\{Z_{1} = {Z_{4} = 0}} & \left\lbrack {{Equation}\mspace{14mu} 88} \right\rbrack \\{X_{3} = {{- X_{2}} = \frac{J_{\max} - 1}{6}}} & \left\lbrack {{Equation}\mspace{14mu} 89} \right\rbrack \\{Z_{2} = {Z_{3} = T}} & \left\lbrack {{Equation}\mspace{14mu} 90} \right\rbrack\end{matrix}$

On the other hand, referring to FIG. 37, the coordinate of an objectpoint on the object plane(3731) corresponding to a pixel (I, J) on theprocessed image plane is given by Eq. 91 through Eq. 93.

$\begin{matrix}{X_{I,J} = \left\{ \begin{matrix}{{X_{1} + {3\left( {X_{2} - X_{1}} \right)\frac{\left( {J - 1} \right)}{\left( {J_{\max} - 1} \right)}\mspace{14mu}{when}\mspace{14mu} 1}} \leq J \leq {1 + \frac{\left( {J_{\max} - 1} \right)}{3}}} \\{{X_{2} + {\left( {X_{3} - X_{2}} \right)\frac{\left\{ {{3\left( {J - 1} \right)} - \left( {J_{\max} - 1} \right)} \right\}}{\left( {J_{\max} - 1} \right)}\mspace{14mu}{when}\mspace{14mu} 1} + \frac{\left( {J_{\max} - 1} \right)}{3}} \leq J \leq {1 + \frac{2\left( {J_{\max} - 1} \right)}{3}}} \\{{X_{3} + {\left( {X_{4} - X_{3}} \right)\frac{\left\{ {{3\left( {J - 1} \right)} - {2\left( {J_{\max} - 1} \right)}} \right\}}{\left( {J_{\max} - 1} \right)}\mspace{14mu}{when}\mspace{14mu} 1} + \frac{2\left( {J_{\max} - 1} \right)}{3}} \leq J \leq J_{\max}}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 91} \right\rbrack \\{Z_{I,J} = \left\{ \begin{matrix}{{Z_{1} + {3\left( {Z_{2} - Z_{1}} \right)\frac{\left( {J - 1} \right)}{\left( {J_{\max} - 1} \right)}\mspace{14mu}{when}\mspace{14mu} 1}} \leq J \leq {1 + \frac{\left( {J_{\max} - 1} \right)}{3}}} \\{{{Z_{2}\mspace{14mu}{when}\mspace{14mu} 1} + \frac{\left( {J_{\max} - 1} \right)}{3}} \leq J \leq {1 + \frac{2\left( {J_{\max} - 1} \right)}{3}}} \\{{Z_{3} - {Z_{3}\frac{\left\{ {{3\left( {J - 1} \right)} - {2\left( {J_{\max} - 1} \right)}} \right\}}{\left( {J_{\max} - 1} \right)}\mspace{14mu}{when}\mspace{14mu} 1} + \frac{2\left( {J_{\max} - 1} \right)}{3}} \leq J \leq J_{\max}}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 92} \right\rbrack \\{Y_{I,J} = {I - I_{o}}} & \left\lbrack {{Equation}\mspace{14mu} 93} \right\rbrack\end{matrix}$

FIG. 38 is a processed image plane extracted from the fisheye image inFIG. 5 using such an object plane, and it has been set-up so thatJ_(max)=601 and I_(max)=240. Therefore, it is certain that the objectplane of a multiple viewpoint panoramic image such as given in FIG. 51in reference 9 is an example of an object plane according to the firstembodiment of the present invention.

On the other hand, FIG. 39 is a cross-sectional diagram of a complexobject plane(3931) in the X-Z plane according to the first embodiment ofthe present invention. For this object plane, an area centered at theoptical axis and having a horizontal FOV of 60° is made to follow arectilinear projection scheme, so that a distortion-free normal imagecan be obtained. Simultaneously, two side areas each having a horizontalFOV of 60° are set-up to follow a projection scheme which is somewhatsimilar to a cylindrical projection scheme. Between these twoboundaries, the object planes are smoothly joined and not are folded. Onthe other hand, the left and the right images cannot be considered asgenuine panoramic images. This is because horizontal intervals in theobject plane corresponding to an identical horizontal FOV are not equal.

The shape of the object plane depicted in FIG. 39 can be described bythe following Eq. 94 through Eq. 103.

$\begin{matrix}{\mspace{79mu}{T = \frac{J_{\max} - 1}{\pi + {2\;\tan\frac{\pi}{6}}}}} & \left\lbrack {{Equation}\mspace{14mu} 94} \right\rbrack \\{\mspace{79mu}{X_{3} = {{- X_{2}} = {T\;\tan\frac{\pi}{6}}}}} & \left\lbrack {{Equation}\mspace{14mu} 95} \right\rbrack \\{\mspace{79mu}{X_{4} = {{- X_{1}} = {X_{3} + T}}}} & \left\lbrack {{Equation}\mspace{14mu} 96} \right\rbrack \\{\mspace{79mu}{Z_{1} = {Z_{4} = 0}}} & \left\lbrack {{Equation}\mspace{14mu} 97} \right\rbrack \\{\mspace{79mu}{Z_{2} = {Z_{3} = T}}} & \left\lbrack {{Equation}\mspace{14mu} 98} \right\rbrack \\{\mspace{79mu}{J_{a} = {1 + {T\frac{\pi}{2}}}}} & \left\lbrack {{Equation}\mspace{14mu} 99} \right\rbrack \\{\mspace{79mu}{J_{b} = {1 + {T\frac{\pi}{2}} + {2\; T\;\tan\frac{\pi}{6}}}}} & \left\lbrack {{Equation}\mspace{14mu} 100} \right\rbrack \\{X_{I,J} = \left\{ \begin{matrix}{{X_{2} - {T\;{\cos\left( \frac{J - 1}{T} \right)}\mspace{14mu}{when}\mspace{14mu} 1}} \leq J \leq J_{a}} \\{{X_{2} + {\left( {X_{3} - X_{2}} \right)\left( \frac{J - J_{a}}{J_{b} - J_{a}} \right)\mspace{14mu}{when}\mspace{14mu} J_{a}}} \leq J \leq J_{b}} \\{{X_{3} + {T\;{\sin\left( \frac{J - J_{b}}{T} \right)}\mspace{14mu}{when}\mspace{14mu} J_{b}}} \leq J \leq J_{\max}}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 101} \right\rbrack \\{\mspace{79mu}{Z_{I,J} = \left\{ \begin{matrix}{{T\;{\sin\left( \frac{J - 1}{T} \right)}\mspace{14mu}{when}\mspace{14mu} 1} \leq J \leq J_{a}} \\{{Z_{2}\mspace{14mu}{when}\mspace{14mu} J_{a}} \leq J \leq J_{b}} \\{{T\;{\cos\left( \frac{J - J_{b}}{T} \right)}\mspace{14mu}{when}\mspace{14mu} J_{b}} \leq J \leq J_{\max}}\end{matrix} \right.}} & \left\lbrack {{Equation}\mspace{14mu} 102} \right\rbrack \\{\mspace{79mu}{Y_{I,J} = {I - I_{o}}}} & \left\lbrack {{Equation}\mspace{14mu} 103} \right\rbrack\end{matrix}$

FIG. 40 is a complex image extracted from the image in FIG. 5. As can beseen in FIG. 40, all the straight lines are represented as straightlines in the middle area. On the other hand, vertical lines arerepresented as vertical lines on two side areas. Similar to a panoramicimage, an image having a FOV of 180° is shown. On the other hand, FIG.41 is an image extracted from FIG. 17. Comparing with FIG. 18, it can beseen that middle region corresponds to a rectilinear image showingstraight lines as straight lines.

X and Z coordinates of object points on an object plane according to thefirst embodiment of the present invention are given as sole functions ofthe lateral pixel coordinate J on the processed image plane as can beseen in Eq. 104 and Eq. 105.X _(I,J) ≡X(I,J)=X(J)≡X _(J)  [Equation 104]Z _(I,J) ≡Z(I,J)=Z(J)≡Z _(J)  [Equation 105]

Furthermore, the Y coordinate of the object point is given as a solefunction of the longitudinal pixel coordinate I on the processed imageplane as can be seen in Eq. 106.Y _(I,J) ≡Y(I,J)=Y(I)≡Y _(I)  [Equation 106]

On the other hand, in correspondence with the longitude V and thelatitude δ in reference 8, a horizontal azimuth angle ψ and a verticalelevation angle δ can be defined for all pixels in the processed imageplane having video signals as in Eq. 107 and Eq. 108.

$\begin{matrix}{{\psi_{I,J} \equiv {\psi\left( {I,J} \right)}} = {\tan^{- 1}\left( \frac{X_{I,J}}{Z_{I,J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 107} \right\rbrack \\{{\delta_{I,J} \equiv {\delta\left( {I,J} \right)}} = {\tan^{- 1}\left( \frac{Y_{I,J}}{\sqrt{X_{I,J}^{2} + Z_{I,J}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 108} \right\rbrack\end{matrix}$

Here, when the Y-axis direction coincides with the vertical direction,the horizontal azimuth angle coincides with the longitude, and thevertical elevation angle coincides with the latitude. Furthermore,considering Eq. 104 through Eq. 106, the horizontal azimuth angle ψ ischaracterized as being a sole function of the lateral pixel coordinatesJ.

$\begin{matrix}{\psi_{I,J} = {\psi_{J} = {\tan^{- 1}\left( \frac{X_{J}}{Z_{J}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 109} \right\rbrack\end{matrix}$

However, in general, the vertical elevation angle is not given as a solefunction of the longitudinal pixel coordinate I.

$\begin{matrix}{\delta_{I,J} = {\tan^{- 1}\left( \frac{Y_{I}}{\sqrt{X_{J}^{2} + Z_{J}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 110} \right\rbrack\end{matrix}$

Imaging systems corresponding to the first embodiment of the presentinvention are characterized in that the horizontal azimuth angle ψ_(I,J)is given as a monotonic function of the lateral pixel coordinate J, andthe vertical elevation angle δ_(I,J) is given as a monotonic function ofthe longitudinal pixel coordinate I. This can be defined as ageneralized panoramic image.

Furthermore, imaging systems corresponding to the first embodiment ofthe present invention are characterized in that straight lines parallelto the Y-axis are captured as straight lines parallel to the sides ofthe image sensor plane along the longitudinal direction.

Furthermore, object planes in imaging systems corresponding to the firstembodiment of the present invention satisfy the following Eq. 111 andEq. 112 on differentiable regions.

$\begin{matrix}{{\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{J}}{\partial J} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 111} \right\rbrack \\{\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} = 1} & \left\lbrack {{Equation}\mspace{14mu} 112} \right\rbrack\end{matrix}$

Second Embodiment

FIG. 35 is a conceptual diagram of an object plane according to thefirst embodiment of the present invention having a shape of ageneralized cylinder, and the resulting complex images can be consideredas generalized horizontal panoramic images. In contrast to this, compleximaging systems according to the second embodiment of the presentinvention are imaging systems for obtaining generalized verticalpanoramic images.

FIG. 42 shows the cross-sectional diagram of an exemplary object planein the Y-Z plane for obtaining such a generalized vertical panoramicimage. This object plane is comprised of a region(1≦I≦I_(a)) centeredabout an optical axis(4201) for obtaining a distortion-free rectilinearimage, a region(I_(a)≦I≦I_(b)) for obtaining an image similar to acylindrical panoramic image, and a region(I_(b)≦I≦I_(max)) for obtaininga tilted rectilinear image. The shape of such an object plane is givenby Eq. 113 through Eq. 126.

$\begin{matrix}{\mspace{79mu}{c = \frac{I_{\max} - 1}{L_{1} + L_{2} + {T\frac{\pi}{2}} + L_{3}}}} & \left\lbrack {{Equation}\mspace{14mu} 113} \right\rbrack \\{\mspace{79mu}{I_{o} = {1 + {cL}_{1}}}} & \left\lbrack {{Equation}\mspace{14mu} 114} \right\rbrack \\{\mspace{79mu}{I_{a} = {I_{o} + {cL}_{2}}}} & \left\lbrack {{Equation}\mspace{14mu} 115} \right\rbrack \\{\mspace{79mu}{I_{b} = {I_{a} + {{cT}\frac{\pi}{2}}}}} & \left\lbrack {{Equation}\mspace{14mu} 116} \right\rbrack \\{\mspace{79mu}{Y_{1} = {- {cL}_{1}}}} & \left\lbrack {{Equation}\mspace{14mu} 117} \right\rbrack \\{\mspace{79mu}{Y_{2} = 0}} & \left\lbrack {{Equation}\mspace{14mu} 118} \right\rbrack \\{\mspace{79mu}{Y_{3} = {cL}_{2}}} & \left\lbrack {{Equation}\mspace{14mu} 119} \right\rbrack \\{\mspace{79mu}{Y_{4} = {Y_{5} = {Y_{3} + {cT}}}}} & \left\lbrack {{Equation}\mspace{14mu} 120} \right\rbrack \\{\mspace{79mu}{Z_{1} = {Z_{2} = {Z_{3} = {{cT} + {cL}_{3}}}}}} & \left\lbrack {{Equation}\mspace{14mu} 121} \right\rbrack \\{\mspace{79mu}{Z_{4} = {cL}_{3}}} & \left\lbrack {{Equation}\mspace{14mu} 122} \right\rbrack \\{\mspace{79mu}{Z_{5} = 0}} & \left\lbrack {{Equation}\mspace{14mu} 123} \right\rbrack \\{\mspace{79mu}{Y_{I,J} = \left\{ \begin{matrix}{{Y_{1} + {\left( {Y_{3} - Y_{1}} \right)\frac{\left( {I - 1} \right)}{\left( {I_{a} - 1} \right)}\mspace{14mu}{when}\mspace{14mu} 1}} \leq I \leq I_{a}} \\{{Y_{3} + {{cT}\;{\sin\left( \frac{I - I_{a}}{cT} \right)}\mspace{14mu}{when}\mspace{14mu} I_{a}}} \leq I \leq I_{b}} \\{{Y_{4}\mspace{14mu}{when}\mspace{14mu} I_{b}} \leq I \leq I_{\max}}\end{matrix} \right.}} & \left\lbrack {{Equation}\mspace{14mu} 124} \right\rbrack \\{Z_{I,J} = \left\{ \begin{matrix}{{Z_{1}\mspace{14mu}{when}\mspace{14mu} 1} \leq I \leq I_{a}} \\{{Z_{4} + {{cT}\;{\cos\left( \frac{I - I_{a}}{cT} \right)}\mspace{14mu}{when}\mspace{14mu} I_{a}}} \leq I \leq I_{b}} \\{{Z_{4} + {\left( {Z_{5} - Z_{4}} \right)\frac{\left( {I - I_{b}} \right)}{\left( {I_{\max} - I_{b}} \right)}\mspace{14mu}{when}\mspace{14mu} I_{b}}} \leq I \leq I_{\max}}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 125} \right\rbrack \\{\mspace{79mu}{X_{I,J} = {J - J_{o}}}} & \left\lbrack {{Equation}\mspace{14mu} 126} \right\rbrack\end{matrix}$

FIG. 43 is a complex image extracted from FIG. 5 using the object planein FIG. 42, and the parameters are given as I_(max)=240, J_(max)=240,L₁=50, L₂=30, T=50 and L₃=80. In FIG. 43, it can be seen that all thestraight lines are represented as straight lines in the upper and thelower regions. In the middle region, it continuously changes much like apanoramic image.

On the other hand, FIG. 44 is another example of a fisheye image givenin reference 9, and FIG. 45 is a complex image extracted from FIG. 44using the object plane in FIG. 42. Parameters are given as I_(max)=800,J_(max)=600, L₁=200, L₂=100, T=100 and L₃=200. In FIG. 45, the upperpart of the image shows a distortion-free image much like the image froman ordinary camera. Since the lower part of the image shows an image asif a camera is heading downward to the ground, it is convenient toidentify the parking lanes. A complex image of this type can be used fora front or a rear view camera for automobiles as in this example. Whenapplied to a video door phone, newspapers or milks lying in front of thedoor as well as visitors can be advantageously identified.

The shape of an object plane according to the second embodiment of thepresent invention can be defined more generally as follows. As is clearfrom Eq. 127, the X coordinate of object points on an object planeaccording to the second embodiment of the present invention is given asa sole function of the lateral pixel coordinate J on the processed imageplane.X _(I,J) ≡X(I,J)=X(J)≡X _(J)  [Equation 127]

On the other hand, as shown in Eq. 128 and Eq. 129, the Y and the Zcoordinates of the object points are given as sole functions of thelongitudinal pixel coordinate I on the processed image plane.Y _(I,J) ≡Y(I,J)=Y(I)≡Y _(I)  [Equation 128]Z _(I,J) ≡Z(I,J)=Z(I)≡Z _(I)  [Equation 129]

On the other hand, an azimuth angle ψ along the lateral direction and anelevation angle δ along the longitudinal direction can be defined forall pixels on the processed image plane using Eq. 107 and Eq. 108.Considering Eq. 128 and Eq. 129, these can be given as Eq. 130 and Eq.131.

$\begin{matrix}{\psi_{I,J} = {\tan^{- 1}\left( \frac{X_{J}}{Z_{I}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 130} \right\rbrack \\{\delta_{I,J} = {\tan^{- 1}\left( \frac{Y_{I}}{\sqrt{X_{J}^{2} + Z_{I}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 131} \right\rbrack\end{matrix}$

Therefore, in the current embodiment, both the azimuth angle in thelateral direction and the elevation angle in the longitudinal directionare given as functions of the lateral pixel coordinate J and thelongitudinal pixel coordinate I. However, as in the first embodiment ofthe present invention, the azimuth angle ψ_(I,J) in the lateraldirection is given as a monotonic function of the lateral pixelcoordinate J, and the elevation angle δ_(I,J) in the longitudinaldirection is given as a monotonic function of the longitudinal pixelcoordinate I.

In imaging systems according to the second embodiment of the presentinvention, straight lines parallel to the X-axis are captured on theprocessed image plane as straight lines parallel to the sides of theimage sensor plane along the lateral direction. Furthermore, objectplane of an imaging system according to the second embodiment of thepresent invention satisfy Eq. 132 and Eq. 133 on differentiable regions.

$\begin{matrix}{\left( \frac{\partial X_{J}}{\partial J} \right)^{2} = 1} & \left\lbrack {{Equation}\mspace{14mu} 132} \right\rbrack \\{{\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I}}{\partial I} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 133} \right\rbrack\end{matrix}$

On the other hand, the equations describing the coordinate of theimaginary image point on the uncorrected image plane corresponding to apixel coordinate (I, J) on the processed image plane of the imagingsystem are identical to those given in the first embodiment.

Third Embodiment

Imaging systems according to the first embodiment of the presentinvention provide generalized horizontal panoramic images, and imagingsystems according to the second embodiment of the present inventionprovide generalized vertical panoramic images. However, a processedimage plane and an object plane can be designed so that a single imagecontains both a generalized horizontal panoramic image and a verticalpanoramic image.

FIG. 46 shows an exemplary structure of a processed image plane(4635)according to the third embodiment of the present invention. Theprocessed image plane(4635) has pixels arranged in a form of atwo-dimensional array with I_(max) rows and J_(max) columns. Thesepixels on the processed image plane are composed of live pixels havingvideo signals and dormant pixels not having video signals. In FIG. 46,pixels belong to sub-regions(4635L, 4635F, 4635R, 3635T, 4635B) on theprocessed image plane(4635) are live pixels, and pixels belong to therest of the region(4635D) are dormant pixels.

In FIG. 46, the location of the optical axis is given as (I_(o), J_(o)),and the sub-region(4635F) containing the optical axis displays arectilinear image showing the scene right in front of the camera withoutany distortion. Furthermore, sub-regions having identification numbers4635L, 4635F and 4635R show a generalized horizontal panoramic image,and the structure can take a relatively arbitrary shape including theshapes of object planes shown in FIG. 37 or in FIG. 39. Furthermore,sub-regions having identification numbers 4635F, 4635T, and 4635B show ageneralized vertical panoramic image, and the structure can take arelatively arbitrary shape including the shape of the object plane shownin FIG. 42. On the other hand, the sub-region having an identificationnumber 4635D does not display any image.

FIG. 47 shows a specific structure of an imaginary object plane(4731).Along the lateral direction, it has an object plane(4731L, 4731F, 4731R)in a form of three viewpoint panoramic image as displayed in FIG. 37,and along the longitudinal direction, it has an object plane(4731F,4731T, 4731B) as displayed in FIG. 42. FIG. 48 shows a complex imageextracted from FIG. 44 using such an object plane. In the regioncorresponding to the identification number 4731F, a distortion-freerectilinear image of the scene in front of the camera is shown. If thiskind of camera is employed as a front view or a rearview camera forautomobiles, then using images of this region, the front side or therear side of an automobile can be checked out while driving theautomobile. On the other hand, since images with identification numbers4731L, 4731F and 4731R constitute a multiple viewpoint panoramic image,obstacles on either side of a car can be conveniently checked out whileparking the car or when coming out of a narrow alley. On the other hand,using the image corresponding to the sub-region with an identificationnumber 4731B, parking lanes can be easily identified while parking thecar. On the other hand, since images corresponding to identificationnumbers 4731F, 4731T and 4731B constitute a generalized verticalpanoramic image, obstacles such as passer-bys on either the front or therear side of a car can be easily identified as demonstrated in FIG. 48.

FIG. 49 shows the dependence of the Z coordinate of the said objectpoint on the pixel coordinate. Referring to FIG. 47 and FIG. 49, in thesub-region having an identification number 4731F, the Z coordinate isgiven as a constant. This can be considered as the simplest function(i.e., a constant function) of the lateral pixel coordinate J, and itcan also be considered as the simplest function of the longitudinalpixel coordinate I. In sub-regions having pixel coordinates 4731L and4731R, the Z coordinate of the said object points is given as a solefunction of the lateral pixel coordinate J. Furthermore, in sub-regionshaving identification numbers 4731T and 4731B, the Z coordinate of thesaid object points is given as a sole function of the longitudinal pixelcoordinate I. In other words, the Z coordinate of any one object pointamong the object points corresponding to the said live pixels depends oneither the lateral pixel coordinate J or the longitudinal pixelcoordinate I, but not simultaneously on both pixel coordinates. On theother hands, the X coordinate of all the object points corresponding tothe said live pixels is given as a sole function of the lateral pixelcoordinate J, and the Y coordinate is given as a sole function of thelongitudinal pixel coordinate I.

A complex image acquisition device according to the third embodiment ofthe present invention simultaneously providing a generalized horizontalpanoramic image and a generalized vertical panoramic image such as thishas the following characteristics. Such a complex image acquisitiondevice is comprised of an image acquisition means for acquiring anuncorrected image plane using a camera equipped with a wide-angle lensrotationally symmetric about an optical axis, an image processing meansfor generating a processed image plane based on the said uncorrectedimage plane, and an image display means for displaying the saidprocessed image plane. The said uncorrected image plane is atwo-dimension array with K_(max) rows and L_(max) columns, and the saidprocessed image plane is a two-dimension array with I_(max) rows andJ_(max) columns. The video signal value of a live pixel on the processedimage plane having a pixel coordinate (I, J) is given by the videosignal value of an imaginary image point on the uncorrected image planeoriginating from an incident ray coming from an imaginary object pointon an imaginary object plane in the world coordinate system having acoordinate (X_(I,J), Y_(I,J), Z_(I,J))≡(X(I, J), Y(I, J), Z(I, J)). TheX coordinate of the said object point in the world coordinate system isgiven as a sole function of the lateral pixel coordinate J as in Eq.134, and the Y coordinate of the said object point is given as a solefunction of the longitudinal pixel coordinate I as in Eq. 135.X _(I,J) ≡X(I,J)=X(J)≡X _(J)  [Equation 134]Y _(I,J) ≡Y(I,J)=Y(I)≡Y _(I)  [Equation 135]

Furthermore, the Z coordinate of any one object point among the saidobject points is given as a sole function of the lateral pixelcoordinate J as in Eq. 136, or it is given as a sole function of thelongitudinal pixel coordinate I as in Eq. 137.Z _(I,J) ≡Z(I,J)=Z(J)≡Z _(J)  [Equation 136]Z _(I,J) ≡Z(I,J)=Z(I)≡Z _(I)  [Equation 137]

Furthermore, the azimuth angle of the said object point in the lateraldirection given by the following Eq. 138 is a monotonic function of thelateral pixel coordinate J, and the elevation angle of the said objectpoint in the longitudinal direction given by the following Eq. 139 is amonotonic function of the longitudinal pixel coordinate I.

$\begin{matrix}{\psi_{I,J} = {\tan^{- 1}\left( \frac{X_{J}}{Z_{I,J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 138} \right\rbrack \\{\delta_{I,J} = {\tan^{- 1}\left( \frac{Y_{I}}{\sqrt{X_{J}^{2} + Z_{I,J}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 139} \right\rbrack\end{matrix}$

Furthermore, the said object plane satisfies Eq. 140 and Eq. 141 ondifferentiable regions.

$\begin{matrix}{{\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial J} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 140} \right\rbrack \\{{\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial I} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 141} \right\rbrack\end{matrix}$

Fourth Embodiment

The shape of an imaginary object plane employed in a panoramic imagingsystem following a cylindrical projection scheme is given by thefollowing Eq. 142 through Eq. 146. Here, ψ₁ is a horizontal incidenceangle (i.e., the longitude) of an incident ray originating from anobject point having a lateral pixel coordinate J=1, and ψ_(Jmax) is ahorizontal incidence angle of an incident ray originating from an objectpoint having a lateral pixel coordinate J=J_(max).

$\begin{matrix}{\rho = {\frac{J_{\max} - 1}{\psi_{J_{\max}} - \psi_{1}} \equiv \frac{J_{\max} - 1}{\Delta\psi}}} & \left\lbrack {{Equation}\mspace{14mu} 142} \right\rbrack \\{\psi_{I,J} = \frac{J - J_{o}}{\rho}} & \left\lbrack {{Equation}\mspace{14mu} 143} \right\rbrack \\{X_{I,J} = {\rho\;{\sin\left( \frac{J - J_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 144} \right\rbrack \\{Z_{I,J} = {\rho\;{\cos\left( \frac{J - J_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 145} \right\rbrack \\{Y_{I,J} = {I - I_{o}}} & \left\lbrack {{Equation}\mspace{14mu} 146} \right\rbrack\end{matrix}$

Such a shape of the object plane satisfies mathematical equations givenin Eq. 111 and Eq. 112. On the other hand, an object plane employed in apanoramic imaging system following an equi-rectangular projection schemesatisfies Eq. 142 through Eq. 145 and the following Eq. 147.

$\begin{matrix}{Y_{I,J} = {\rho\;{\tan\left( \frac{I - I_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 147} \right\rbrack\end{matrix}$

Furthermore, an object plane employed in a panoramic imaging systemfollowing a Mercator projection scheme satisfies Eq. 142 through Eq. 145and the following Eq. 148.

$\begin{matrix}{Y_{I,J} = {\rho\;{\sinh\left( \frac{I - I_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 148} \right\rbrack\end{matrix}$

The X and the Z coordinates of object points on object planes followingthese cylindrical projection scheme, equi-rectangular projection scheme,or Mercator projection scheme are given as sole functions of the lateralpixel coordinate J, and the Y coordinate is given as a sole function ofthe longitudinal pixel coordinate I. Furthermore, object planesfollowing these Mercator projection scheme or equi-rectangularprojection scheme do not have simple geometrical meanings such as theone given in the first embodiment of the present invention. In otherwords, Eq. 111 or Eq. 149 is satisfied along the lateral direction, butthere is no corresponding equation for the longitudinal direction.

$\begin{matrix}{{\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{J}}{\partial J} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 149} \right\rbrack \\{\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} \neq 1} & \left\lbrack {{Equation}\mspace{14mu} 150} \right\rbrack\end{matrix}$

Shapes of object planes employed in panoramic imaging systems providingthe most general horizontal panoramic images including the one given inthe first embodiment of the present invention are obtained by removingthe constraints given by Eq. 111 and Eq. 112. In other words, the shapeof the object plane in the fourth embodiment is obtained by setting-upthe three-dimensional coordinate (X(I, J), Y(I, J), Z(I, J)) of theobject point Q in the world coordinate system corresponding to eachpixel (I, J) on the processed image plane. A generalized horizontalpanoramic image can be obtained using Eq. 66 through Eq. 77 except forEq. 71 and Eq. 72, or Eq. 111 and Eq. 112, for each and every objectpoint.

In the same manner, for an imaging system providing a generalizedvertical panoramic image, or a complex image simultaneously containing ageneralized horizontal panoramic image and a generalized verticalpanoramic image, the most general imaging system can be realized byremoving the constraints given by Eq. 140 and Eq. 141 in the thirdembodiment. However, for the provided image to be a desirable one,several conditions such as provided in the third embodiment of thepresent invention must be satisfied.

Fifth Embodiment

In the first through the fourth embodiments, image processing methodsfor obtaining processed image planes corresponding to general shapes ofobject planes are provided. However, in a case where the object planedoes not have a simple geometrical meaning as in the fourth embodiment,an image processing method which does not require an imaginary objectplane can be simpler. In this case, a zenith angle θ and an azimuthangle φ of an incident ray can be directly set-up for every live pixel(I, J) on the processed image plane. Therefore, as in Eq. 151 and Eq.152, a zenith angle θ and an azimuth angle φ of an incident ray can bedirectly set-up for every live pixel (I, J) on the processed imageplane.θ_(I,J)=θ(I,J)  [Equation 151]φ_(I,J)=φ(I,J)  [Equation 152]

Using the real projection scheme of the fisheye lens given by Eq. 66 andthe magnification ratio g of the image given by Eq. 67, the pixelcoordinate of an imaginary image point on the uncorrected image planecorresponding to such zenith and azimuth angles of an incident ray canbe obtained using Eq. 153 through Eq. 154.x′ _(I,J) =L _(o) +gr(θ_(I,J))cos φ_(I,J)  [Equation 153]y′ _(I,J) =K _(o) +gr(θ_(I,J))sin φ_(I,J)  [Equation 154]

A fisheye image following a stereographic projection scheme can beconsidered as a useful example of the fifth embodiment. Most of thefisheye lenses are manufactured as to follow equidistance projectionschemes. However, the real projection scheme of a fisheye lens shows acertain amount of error with an ideal equidistance projection scheme. Onthe other hand, a fisheye image that appears most natural to the nakedeye is a fisheye image following a stereographic projection scheme, andthe image height of a fisheye image following a stereographic projectionscheme is given by Eq. 155.

$\begin{matrix}{{r^{''}(\theta)} = {\frac{r_{2}^{''}}{\tan\left( \frac{\theta_{2}}{2} \right)}{\tan\left( \frac{\theta}{2} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 155} \right\rbrack\end{matrix}$

Here, r″(θ) is an image height on the processed image planecorresponding to the zenith angle θ measured in pixel distances. Fromnow on, to distinguish between the physical image height on an imagesensor plane and the image height measured in pixel distances, theformer will be referred to as an image height and the latter will bereferred to as a pixel distance. On the other hand, θ₂ is a referencezenith angle in a fisheye image, and r″₂ is a pixel distancecorresponding to this reference zenith angle. In general, the maximumzenith angle in a fisheye image can be set-up as a reference zenithangle, but it is not a mandatory requirement. On the other hand, a pixeldistance r″_(I,J) corresponding to a pixel (I, J) is given by Eq. 156.r″ _(I,J) ≡r″(θ_(I,J))=√{square root over ((I−I _(o))²+(J−J_(o))²)}{square root over ((I−I _(o))²+(J−J _(o))²)}  [Equation 156]

Therefore, the zenith angle of an incident ray corresponding to a pixeldistance r″_(I,J) on a fisheye image following a stereographicprojection scheme is given by Eq. 157.

$\begin{matrix}{{\theta\left( {I,J} \right)} = {2{\tan^{- 1}\left\lbrack {\frac{\tan\left( \frac{\theta_{2}}{2} \right)}{r_{2}^{''}}r_{I,J}^{''}} \right\rbrack}}} & \left\lbrack {{Equation}\mspace{14mu} 157} \right\rbrack\end{matrix}$

On the other hand, the azimuth angle of the incident ray is given by Eq.158.

$\begin{matrix}{{\phi\left( {I,J} \right)} = {\tan^{- 1}\left( \frac{I - I_{o}}{J - J_{o}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 158} \right\rbrack\end{matrix}$

FIG. 50 is a stereographic fisheye image extracted from the fisheyeimage in FIG. 5. Such an imaging system provides desirable fisheyeimages when fisheye images have to be observed with bare eyes. On theother hand, FIG. 51 is another interior scene captured using a fisheyelens having a FOV of 190° installed on an interior ceiling, and FIG. 52is a stereographic fisheye image extracted from FIG. 51. The parametersare given as I_(max)=J_(max)=800, r″₂=J_(max)−J_(o), and θ₂=90°.

On the other hand, an ideal equidistance projection scheme is given byEq. 159.

$\begin{matrix}{{r^{''}(\theta)} = {\frac{r_{2}^{''}}{\theta_{2}}\theta}} & \left\lbrack {{Equation}\mspace{14mu} 159} \right\rbrack\end{matrix}$

Therefore, in order to obtain a fisheye image following a mathematicallyprecise equidistance projection scheme from a fisheye image exhibiting acertain amount of error with an ideal equidistance projection scheme,the zenith angle of an incident ray corresponding to a pixel coordinate(I, J) on the processed image plane can be obtained using Eq. 160, andthe azimuth angle can be obtained using Eq. 158.

$\begin{matrix}{{\theta\left( {I,J} \right)} = {\frac{\theta_{2}}{r_{2}^{''}}r_{I,J}^{''}}} & \left\lbrack {{Equation}\mspace{14mu} 160} \right\rbrack\end{matrix}$

FIG. 53 is a fisheye image following an equidistance projection schemeextracted from the fisheye image in FIG. 51. Such an imaging system canbe used to obtain precise all sky images in special areas such asstellar observatory or air traffic control.

Sixth Embodiment

In the fifth embodiment, a zenith angle θ and an azimuth angle φ havebeen directly set-up for every live pixel (I, J) on the processed imageplane. However, setting-up the zenith angles and the azimuth angles ofincident rays corresponding to a desirable complex image including apanoramic image can be considerably difficult. In a complex imagesimilar to a panoramic image, the zenith angle and the azimuth angle ofan incident rays corresponding to such a complex image may not beintuitively obvious, but the horizontal incident angle (i.e., theazimuth angle) ψ and the vertical incident angle (i.e., the elevationangle) δ may be definable in an obvious manner. Therefore, an imageprocessing algorithm given in Eq. 161 and Eq. 162 can be more desirablewhich sets up the horizontal incidence angle ψ and the verticalincidence angle δ of an incident ray corresponding to each and everypixel (I, J) on the processed image plane.ψ_(I,J)=ψ(I,J)  [Equation 161]δ_(I,J)=δ(I,J)  [Equation 162]

From the horizontal incidence angle ψ and the vertical incidence angle δof an incident ray, the azimuth angle of the incident ray can beobtained from Eq. 163 and the zenith angle of the incident ray can beobtained from Eq. 164.

$\begin{matrix}{\phi_{I,J} = {\tan^{- 1}\left( \frac{\tan\;\delta_{I,J}}{\sin\;\psi_{I,J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 163} \right\rbrack\end{matrix}$θ_(I,J)=cos⁻¹(cos δ_(I,J) cos ψ_(I,J))  [Equation 164]

The coordinate of the image point on the uncorrected image planecorresponding to these azimuth angle and zenith angle of an incident raycan be obtained using Eq. 165 and Eq. 166.x′ _(I,J) =L _(o) +gr(θ_(I,J))cos φ_(I,J)  [Equation 165]y′ _(I,J) =K _(o) +gr(θ_(I,J))sin φ_(I,J)  [Equation 166]

FIG. 54 is a complex image having a horizontal incidence angle ψ givenby Eq. 167 and a vertical incidence angle δ given by Eq. 168 extractedfrom the fisheye image in FIG. 5.

$\begin{matrix}{{\psi(J)} = {\pi\left\{ {\frac{J - 1}{J_{\max} - 1} - \frac{1}{2}} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 167} \right\rbrack \\{{\delta(I)} = {\frac{\pi}{2}\left\{ {\frac{I - 1}{I_{\max} - 1} - \frac{1}{2}} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 168} \right\rbrack\end{matrix}$

For such an imaging system, the desired purpose can be accomplished byforcing the field of view of the image displayed on the monitor screen,or in other words, the angular range of the incident rays correspondingto live pixels on the processed image plane. For example, although thelateral size of the processed image plane is the same as thelongitudinal size in FIG. 54, the represented FOV is 180° along thelateral direction, and 90° along the longitudinal direction.

In these algorithms, the horizontal incidence angle and the verticalincidence angle can be given as arbitrary functions of the lateral pixelcoordinate J and the longitudinal pixel coordinate I. However, compleximages obtained under these conditions may not be desirable. To obtaindesirable images along the line of panoramic images, it will bedesirable that the horizontal incidence angle is given as a solefunction of the lateral pixel coordinate J as in Eq. 167, and thevertical incidence angle is given as a sole function of the longitudinalpixel coordinate I as in Eq. 168.

Seventh Embodiment

The first through the sixth embodiments of the present invention providecomplex image acquisition devices using image processing algorithms forobtaining desirable complex images. However, if it is not necessary tochange the projection scheme of the processed image plane as in digitalpan•tilt technology, and the demand for the imaging system is large,then pixel disposition in a CMOS image sensor plane can be changed sothat a desirable complex image can be obtained without additional imageprocessing. This case can be considered as a hard-wired imageprocessing. In this case, the CMOS image sensor must be manufactured asa set with the wide-angle lens that will be used together.

The center coordinate (x_(I,J), y_(I,J)) of each pixel on an imagesensor plane of a CMOS image sensor for obtaining horizontal panoramicimages without additional software image processing can be obtained asfollows. First, if the CMOS image sensor has I_(max) rows and J_(max)columns, and if it is desired to obtain a panoramic image having ahorizontal FOV of Δψ, then the radius of the imaginary object plane isgiven by Eq. 169.

$\begin{matrix}{\rho = \frac{J_{\max} - 1}{\Delta\psi}} & \left\lbrack {{Equation}\mspace{14mu} 169} \right\rbrack\end{matrix}$

Here, the horizontal FOV Δψ of the said panoramic image must be largerthan 0 and smaller than 2π.

A CMOS image sensor can exist as a commodity independent from a fisheyelens. In a CMOS image sensor separated from a fisheye lens, the locationof the optical axis is a reference point on the image sensor plane ofthe CMOS image sensor, and this reference point is used as the origin ofthe first rectangular coordinate system describing the center positionsof pixels. Furthermore, the real projection scheme of a fisheye lens ora wide-angle lens can be considered as a monotonically increasingfunction r=r(θ) of an angle θ passing through the origin.

If the location of the optical axis of a fisheye lens, that is usedalong with the CMOS image sensor, on the image sensor plane of the CMOSimage sensor corresponds to a lateral pixel coordinate J_(o) and alongitudinal pixel coordinate I_(o), then the horizontal incidence angleof an incident ray impinging on the pixel P(I, J) having a lateral pixelcoordinate J and a longitudinal pixel coordinate I must be given by Eq.170.

$\begin{matrix}{\psi_{J} = \frac{J - J_{o}}{\rho}} & \left\lbrack {{Equation}\mspace{14mu} 170} \right\rbrack\end{matrix}$

Furthermore, the coordinate of an object point on the imaginary objectplane corresponding to this pixel is given by Eq. 171 through Eq. 173.

$\begin{matrix}{X_{J} = {\rho\;{\sin\left( \frac{J - J_{o}}{\rho} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 171} \right\rbrack \\{Z_{J} = {{\rho cos}\left( \frac{J - J_{o}}{\rho} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 172} \right\rbrack \\{Y_{I} = {F(I)}} & \left\lbrack {{Equation}\mspace{14mu} 173} \right\rbrack\end{matrix}$

Here, F(I) is a monotonically increasing function of the longitudinalpixel coordinate I passing through the origin, and it must satisfy Eq.174 and Eq. 175.

$\begin{matrix}{{F(I)} = 0} & \left\lbrack {{Equation}\mspace{14mu} 174} \right\rbrack \\{\frac{\partial{F(I)}}{\partial I} > 0} & \left\lbrack {{Equation}\mspace{14mu} 175} \right\rbrack\end{matrix}$

If the panoramic image follows a cylindrical projection scheme, then thesaid function F(I) is given by Eq. 176.F(I)=I−I _(o)  [Equation 176]

On the other hand, if the panoramic image follows an equi-rectangularprojection scheme, then the said function F(I) is given by Eq. 177, andif the panoramic image follows a Mercator projection scheme, then thesaid function F(I) is given by Eq. 178.

$\begin{matrix}{{F(I)} = {{\rho tan}\left( \frac{I - I_{o}}{\rho} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 177} \right\rbrack \\{{F(I)} = {{\rho sinh}\left( \frac{I - I_{o}}{\rho} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 178} \right\rbrack\end{matrix}$

On the other hand, the center coordinate of a pixel on the said imagesensor plane is given as (x_(I,J), y_(I,J)) in the first rectangularcoordinate system. The said first rectangular coordinate system takesthe said location of the optical axis as the reference point, and takesan axis that passes through the said reference point and is parallel tothe sides of the image sensor plane along the lateral direction as thex-axis. The positive(+) x-axis runs from the left side of the said imagesensor plane to the right side. Furthermore, an axis that passes throughthe said reference point and is parallel to the sides of the imagesensor plane along the longitudinal direction is taken as the y-axis.The positive(+) y-axis runs from the top end of the said image sensorplane to the bottom end. The relations among the image sensor plane, theobject plane, the world coordinate system, and the first rectangularcoordinate system can be easily understood with reference to FIG. 35.

However, there is one point that has to be kept in mind. When an objectin the world coordinate system forms an image on the CMOS image sensorplane by the image forming properties of the wide-angle lens, theresulting image is an upside-down image. Since an upside-down image isdifficult to be recognized, the image is deliberately inverted again sothat the direction of the image appears normal. Therefore, it has to beunderstood that a process for inverting the top and the bottom of animage has been implicitly taken in the coordinate systems of FIG. 35.

The coordinate of the said reference point in this first rectangularcoordinate system is (0, 0), and the center coordinate of the said pixelis given by the following Eq. 179 through Eq. 183.

$\begin{matrix}{\phi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I}}{X_{J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 179} \right\rbrack \\{\theta_{I,J} = {\cos^{- 1}\left( \frac{Z_{J}}{\sqrt{X_{J}^{2} + Y_{I}^{2} + Z_{J}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 180} \right\rbrack \\{r_{I,J} = {r\left( \theta_{I,J} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 181} \right\rbrack \\{x_{I,J} = {r_{I,J}\cos\;\phi_{I,J}}} & \left\lbrack {{Equation}\mspace{14mu} 182} \right\rbrack \\{y_{I,J} = {r_{I,J}\sin\;\phi_{I,J}}} & \left\lbrack {{Equation}\mspace{14mu} 183} \right\rbrack\end{matrix}$

Here, r(θ) is a monotonically increasing function of an angle θ passingthrough the origin and it must satisfy Eq. 184 and Eq. 185.

$\begin{matrix}{{r(0)} = 0} & \left\lbrack {{Equation}\mspace{14mu} 184} \right\rbrack \\{\frac{\partial{r(\theta)}}{\partial\theta} > 0} & \left\lbrack {{Equation}\mspace{14mu} 185} \right\rbrack\end{matrix}$

Specifically, r(θ) is the real projection scheme of a wide-angle lensthat is used along with the CMOS image sensor, and is an image height rgiven as a function of the zenith angle θ of an incident ray.

FIG. 55 is the fisheye image in FIG. 5 overlapped with the locations ofpixels necessary to obtain a panoramic image having a horizontal FOV of180° and following a cylindrical projection scheme from the fisheyeimage in FIG. 5 following an equidistance projection scheme. Here, ithas been assumed that J_(max)=16 and I_(max)=12, which correspond to aVGA-grade image having 480 rows and 640 columns with a width:heightratio of 4:3 reduced to 1/40 of the original size. On the other hands,FIG. 56 is a diagram where only pixels are drawn. Here, all pixelshaving identical column number J are lying on a curve which is stretchedalong the longitudinal direction, and likewise, all pixels havingidentical row number I are lying on a curve which is stretched along thelateral direction.

FIG. 57 is a diagram for understanding the pixel arrangements in FIG. 56in more details. Such a CMOS image sensor for panoramic imaging systemincludes within its sensor plane(5713) a multitude of pixels given in aform of an array having I_(max) rows and J_(max) columns. Said CMOSimage sensor has a reference point having a row number I_(o) and acolumn number J_(o), and this reference point is the location of theoptical axis of a fisheye lens that is used along with the CMOS imagesensor. Since the location of the optical axis does not have to coincidewith the position of any one pixel, the said row number I_(o) is a realnumber larger than 1 and smaller than I_(max), and the said columnnumber J_(o) is a real number larger than 1 and smaller than J_(max).However, the row number I and the column number J of any one pixel P(I,J) are both natural numbers.

The location of any one pixel is described by the first rectangularcoordinate system, wherein an axis that passes through the saidreference point O and is parallel to the sides of the image sensor planealong the lateral direction is taken as the x-axis. The positive(+)x-axis runs from the left side of the said image sensor plane to theright side. Likewise, an axis that passes through the said referencepoint and is parallel to the sides of the image sensor plane along thelongitudinal direction is taken as the y-axis. The positive(+) y-axisruns from the top end of the image sensor plane to the bottom end.Therefore, the coordinate of the said reference point is given by (0,0), and the center coordinate of a pixel P(I, J) having a row number Iand a column J are given as (x_(I,J), y_(I,J)). It has to be kept inmind that two kinds of coordinates are used here. One is a pixelcoordinate measured in pixel units in a digitized digital image, andanother is a physical coordinate describing the location of a real pixelin the image sensor plane. A photodiode in an actual pixel has a finitearea, and a position corresponding to the center of mass of thisphotodiode is considered as the center coordinate of the pixel. Forconvenience, the center coordinate of a pixel can be described in unitof millimeters. However, it has to be reminded that a pixel coordinatehas no unit. Furthermore, although the arrangement of the centerpositions of the real pixels in FIG. 57 looks like a spool, it is anicely ordered matrix in the viewpoint of a digital image much like achess board.

Recently, technology developments are actively underway in order toincrease the dynamic range of CMOS image sensors, and such image sensorsare referred by various names such as HDR(High Dynamic Range) or WDR(Wide Dynamic Range) sensors. Various methods are used to realize suchtechnologies. As one method, for example, every pixel is comprised of alarger photodiode and a smaller photodiode, and dynamic range can beincreased by proper image processing. In other words, an image solelyobtained from photodiodes with smaller areas will be adequately exposedto bright objects, and an image solely obtained from photodiodes withlarger areas will be adequately exposed to dark objects. Therefore, bymixing these two images, an image can be obtained that is adequatelyexposed simultaneously to bright objects and to dark objects. In a caseusing such technologies, the center coordinate of a pixel according tothe current invention must be interpreted as the center of mass ofentire photodiode including the photodiode with a larger area and thephotodiode with a smaller area.

When pixels having an identical row number I are all joined together byconnecting adjacent pixels with line segments, the overall shape becomesa curve H(I) passing all the said pixels that is substantially stretchedalong the lateral direction. Likewise, when pixels having an identicalcolumn number J are all joined together by connecting adjacent pixelswith line segments, the overall shape becomes a curve V(J) passing allthe said pixels that is substantially stretched along the longitudinaldirection. Pixels having an identical row number are simultaneouslyaccessed by the vertical shift register, and pixels having an identicalcolumn number are simultaneously accessed by the horizontal shiftregister.

In a CMOS image sensor for obtaining such horizontal panoramic images,pixels P(I, J<J_(o)) belonging to a column with the column number Jsmaller than J_(o) form a curve V(J<J_(o)) open toward the positivex-axis, and pixels P(I, J>J_(o)) belonging to a column with the columnnumber J larger than J_(o) form a curve V(J>J_(o)) open toward thenegative x-axis. If the column number J_(o) of the optical axis is givenas a natural number, then pixels P(I, J_(o)) belonging to a column withthe column number J equal to J_(o) form not a curve but a vertical line.

Likewise, pixels P(I<I_(o), J) belonging to a row with the row number Ismaller than I_(o) form a curve H(I<I_(o)) open toward the negativey-axis, and pixels P(I>I_(o), J) belonging to a row with the row numberI larger than I_(o) form a curve open toward the positive y-axis. If therow number I_(o) of the optical axis is given as a natural number, thenpixels P(I_(o), J) belonging to a row with 1 equal to I_(o) form not acurve but a horizontal line.

FIG. 58 is a schematic diagram for clearly understanding the notion of acurve open toward one direction in the current invention. On the imagesensor plane in FIG. 58, the said reference point lies at theintersection point between the x-axis and the y-axis, and a hypotheticalpixel coordinate for the reference point is (I_(o), J_(o)). When pixelshaving a column number J smaller than J_(o) are all joined together byconnecting neighboring pixels with line segments, it takes the shape ofa curve substantially stretched along the longitudinal direction. Asmooth curve passing all the pixels, such as a Spline curve, will becalled as V(J). Roughly speaking, the left side L of the curve can bedifferentiated from the right side R with reference to this curve.

In FIG. 58, since the column number J is smaller than J_(o), the saidreference point (I_(o), J_(o)) lies at the right side R of the saidcurve. The reference point can be equally described as to lie at thepositive(+) x-axis direction. On the other hand, arbitrary two pixelsP(I₁, J) and P(I₂, J) lying on the said curve V(J) are considered. Thefirst tangent line t(I₁, J) passing through the first pixel P(I₁, J) andthe tangent line t(I₂, J) passing through the second pixel P(I₂, J) aredrawn in FIG. 58. Furthermore, the first normal n(I₁, J) passing throughthe first pixel and perpendicular to the first tangent line, and thesecond normal n(I₂, J) passing through the second pixel andperpendicular to the second tangent line are also drawn.

The first normal and the second normal may or may not have anintersection point. For example, in an ordinary CMOS image sensor, thesaid curve V(J) is a straight line along the longitudinal direction, andconsequently the first tangent line and the second tangent line areparallel to said curve V(J), and the first normal and the second normalare also parallel to each other. Therefore, the first normal and thesecond normal do not have an intersection point no matter how far theyare extended. On the other hand, in a case where said pixels are notlying on a straight line as in FIG. 58, the first normal and the secondnormal share one intersection point C(I₁, I₂, J). In the example of FIG.58, the said intersection point lies at the right side of the said curvemuch like the said reference point, and at the same time, it lies at thepositive x-axis direction. The notion of forming a curve open toward thepositive x-axis in the current invention refers to this case. A curveopen toward the negative x-axis, or a curve open toward the positivey-axis can be understood in the same manner.

On the other hand, FIG. 59 shows a pixel disposition in a CMOS imagesensor for obtaining panoramic images having an equi-rectangularprojection scheme, and FIG. 60 shows a pixel disposition in a CMOS imagesensor for obtaining panoramic images having a Mercator projectionscheme. Comparing FIG. 57, FIG. 59 and FIG. 60, it can be noticed thatstructures of CMOS image sensors for obtaining horizontal panoramicimages are similar to each other.

On the other hand, FIG. 61 shows a pixel disposition in a CMOS imagesensor for obtaining a panoramic image following a cylindricalprojection scheme with 190° horizontal FOV using a fisheye lens from theinvention of a prior art. The CMOS image sensor has been assumed thatthe length of the image sensor plane along the lateral direction is 4.8mm, and the length along the longitudinal direction is 3.6 mm. From FIG.61, it can be seen that the structure has not been significantly changedeven though the calibrated distortion of the said fisheye lens has beentaken into account.

Eighth Embodiment

Provided in the seventh embodiment of the current invention are CMOSimage sensors for obtaining horizontal panoramic images withoutadditional software image processing. Similar to this, the centercoordinate (x_(I,J), y_(I,J)) of each pixel in a CMOS image sensor forobtaining rectilinear images without additional software imageprocessing can be obtained by the following methods. First, assume thatthe CMOS image sensor has I_(max) rows and J_(max) columns, and furtherassume that it is desired to obtain a rectilinear image with ahorizontal FOV of Δψ. Then, the distance s″ from the origin of the worldcoordinate system to the hypothetical object plane is given by thefollowing Eq. 186.

$\begin{matrix}{s^{''} = \frac{J_{\max} - 1}{2{\tan\left( \frac{\Delta\psi}{2} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 186} \right\rbrack\end{matrix}$

Here, due to the mathematical nature of rectilinear images, thehorizontal FOV Δψ of the said rectilinear image must be larger than 0and smaller than π.

Regarding the wide-angle lens that is used as a set with the CMOS imagesensor, let's further assume that the location of the optical axis ofthe wide-angle lens on the image sensor plane corresponds to a lateralpixel coordinate J_(o) and a longitudinal pixel coordinate I_(o). Then,the zenith angle of an incident ray impinging on a pixel P(I, J) havinga lateral pixel coordinate J and a longitudinal pixel coordinate I isgiven by Eq. 187, and the azimuth angle must be given by Eq. 188.

$\begin{matrix}{\theta_{I,J} = {\tan^{- 1}\left\{ \frac{\sqrt{\left( {I - I_{o}} \right)^{2} + \left( {J - J_{o}} \right)^{2}}}{s^{''}} \right\}}} & \left\lbrack {{Equation}\mspace{14mu} 187} \right\rbrack \\{\phi_{I,J} = {\tan^{- 1}\left( \frac{I - I_{o}}{J - J_{o}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 188} \right\rbrack\end{matrix}$

From this, the center coordinate (x_(I,J), y_(I,J)) of the said pixel isgiven by the following Eq. 189 through Eq. 191.r _(I,J) =r(θ_(I,J))  [Equation 189]x _(I,J) =r _(I,J) cos(φ_(I,J))  [Equation 190]y _(I,J) =r _(I,J) sin(φ_(I,J))  [Equation 191]

Here, r(θ) is a monotonically increasing function of the angle θ passingthrough the origin and must satisfy Eq. 192 and Eq. 193.

$\begin{matrix}{{r(0)} = 0} & \left\lbrack {{Equation}\mspace{14mu} 192} \right\rbrack \\{\frac{\partial{r(\theta)}}{\partial\theta} > 0} & \left\lbrack {{Equation}\mspace{14mu} 193} \right\rbrack\end{matrix}$

Specifically, r(θ) is the real projection scheme of the wide-angle lensthat is used along with the CMOS image sensor, and is an image height rgiven as a function of the zenith angle θ of the incident ray.

FIG. 62 is a fisheye image in FIG. 5 overlapped with the pixel locationsnecessary to obtain wide-angle images following a rectilinear projectionscheme and having a horizontal FOV of 120° from the fisheye image inFIG. 5 following an equidistance projection scheme. Here, it has beenassumed that J_(max)=16 and I_(max)=12, and the reference point lies atthe center of the image sensor plane. On the other hand, FIG. 63 is adiagram showing only the pixels P(I, J). Pixels having an identicalcolumn number J are lying on a curve along the longitudinal direction.Likewise, pixels having an identical row number I are lying on a curvealong the lateral direction.

In such a CMOS image sensor for obtaining rectilinear images, pixelsP(I, J<J_(o)) belonging to a column with the column number J smallerthan J_(o) form a curve open toward the positive x-axis, and pixels P(I,J>J_(o)) belonging to a column with the column number J larger thanJ_(o) form a curve open toward the negative x-axis. Likewise, pixelsP(I<I_(o), J) belonging to a row with the row number I smaller thanI_(o) form a curve open toward the positive y-axis, and pixelsP(I>I_(o), J) belong to a row with the row number I larger than I_(o)form a curve open toward the negative y-axis.

Ninth Embodiment

The first through the sixth embodiments of the current invention providecomplex image acquisition devices using image processing algorithms forobtaining desirable complex images. Also, in these cases, if there arelarge demands for imaging systems, pixel dispositions in CMOS imagesensor planes can be altered to obtain desirable complex images withoutseparate image processing stages.

In this case, the size of the processed image plane, in other words, thenumber of pixels (I_(max), J_(max)) in the CMOS image sensor isdetermined, and then a projection scheme for all the pixels (I, J) onthe said image sensor plane are determined. In other words, zenithangles and azimuth angles of incident rays given by Eq. 151 and Eq. 152can be assigned or horizontal incident angles in the lateral directionand vertical incident angles in the longitudinal direction given by Eq.161 and Eq. 162 can be assigned. Or, shapes of the object planes can beassigned as in the first through the fourth embodiments of the currentinvention.

For example, a CMOS image sensor corresponding to the first embodimentof the current invention is used as a set with a wide-angle imaging lensrotationally symmetric about an optical axis. The said CMOS image sensorhas multitude of pixels in the image sensor plane given in a form of anarray having I_(max) rows and J_(max) columns. The real projectionscheme of the said lens is the image height on the image sensor planeobtained as a function of the zenith angle θ of the correspondingincident ray and given as r=r(θ).

The location of any one pixel in the CMOS image sensor plane isdescribed by the first rectangular coordinate system previouslydescribed. Therefore, the center coordinate of a pixel P(I, J) having arow number I and a column number J is given as (x_(I,J), y_(I,J)). Thiscenter coordinate is the coordinate of a hypothetical image point formedon the said image sensor plane by the said wide-angle lens from anincident ray originating from a hypothetical object point having acoordinate (X_(I,J), Y_(I,J), Z_(I,J))≡(X(I, J), Y(I, J), Z(I, J)) on ahypothetical object plane, and it is given by Eq. 194 through Eq. 202.

$\begin{matrix}{r = {r(\theta)}} & \left\lbrack {{Equation}\mspace{14mu} 194} \right\rbrack \\{X_{I,J} = {X\left( {I,J} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 195} \right\rbrack \\{Y_{I,J} = {Y\left( {I,J} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 196} \right\rbrack \\{Z_{I,J} = {Z\left( {I,J} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 197} \right\rbrack \\{\phi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I,J}}{X_{I,J}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 198} \right\rbrack \\{\theta_{I,J} = {\cos^{- 1}\left( \frac{Z_{I,J}}{\sqrt{X_{I,J}^{2} + Y_{I,J}^{2} + Z_{I,J}^{2}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 199} \right\rbrack \\{r_{I,J} = {r\left( \theta_{I,J} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 200} \right\rbrack \\{x_{I,J} = {r_{I,J}\cos\;\phi_{I,J}}} & \left\lbrack {{Equation}\mspace{14mu} 201} \right\rbrack \\{y_{I,J} = {r_{I,J}\sin\;\phi_{I,J}}} & \left\lbrack {{Equation}\mspace{14mu} 202} \right\rbrack\end{matrix}$

Here, rectangular coordinates given by Eq. 195 through Eq. 197 mustsatisfy Eq. 203 and Eq. 204.

$\begin{matrix}{{\left( \frac{\partial X_{I,J}}{\partial J} \right)^{2} + \left( \frac{\partial Y_{I,J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial J} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 203} \right\rbrack \\{{\left( \frac{\partial X_{I,J}}{\partial I} \right)^{2} + \left( \frac{\partial Y_{I,J}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial I} \right)^{2}} = 1} & \left\lbrack {{Equation}\mspace{14mu} 204} \right\rbrack\end{matrix}$

However, when an object plane is folded, derivatives do not exist onfolded points, and Eq. 203 or Eq. 204 cannot hold true. Therefore, theshapes of such object planes can be summarized as follows. The shape ofan object plane according to the ninth embodiment of the presentinvention, in other words, X=X(I, J), Y=Y(I, J) and Z=Z(I, J), arecontinuous functions of I and J, and satisfy Eq. 203 and Eq. 204 ondifferentiable points.

Using such a method, a CMOS image sensor which does not require separateimage processing can be provided corresponding to any image processingalgorithm given in the first through the sixth embodiments. FIG. 64shows a pixel disposition in a CMOS image sensor for obtaining multipleviewpoint panoramic images corresponding to FIG. 38, and FIG. 65 shows apixel disposition in a CMOS image sensor for obtaining complex panoramicimages corresponding to FIG. 46 through FIG. 49.

Tenth Embodiment

There are two kinds of pixels in FIG. 65, namely sensitive pixels markedas circles and insensitive pixels marked as dots. The pixel P(I₁, J₁)having a row number I₁ and a column number J₁ is an example of sensitivepixels, and the pixel P(I₂, J₂) having a row number I₂ and a columnnumber J₂ is an example of insensitive pixels. Sensitive pixels arepixels normally found in ordinary CMOS image sensors. A sensitive pixelbecomes a live pixel in the current invention when there is video signalin the sensitive pixel, and becomes a dormant pixel when there is novideo signal.

On the other hand, an insensitive pixel in the current invention refersto a pixel where there is no video signal coming out from the pixel, orwhen the video signal is relatively very weak. It is not referring to abad pixel from where no video signal is coming out due to manufacturingerrors in CMOS sensor fabrication process. It rather refers to a casewhere the CMOS sensor is intentionally fabricated so that video signalsare not coming out from the selected pixels. Therefore, the part of animage corresponding to insensitive pixels appear dark black, and it hasthe same effect as can be seen in the lower left and the lower rightcorners in FIG. 48. Therefore, the lower left and the lower rightcorners in FIG. 48 may be due to dormant pixels originating from thelack of video signals in normal sensitive pixels. Or, it may be due tothe fact that corresponding pixels in the CMOS image sensor areinsensitive pixels. Whatever the reason is, there is no differencebetween the two digital images.

FIG. 66 schematically shows a pixel structure in an ordinary CMOS imagesensor. CMOS image sensors are normally produced by forming n-typephotodiodes(n-PD: 6656) on a p-type substrate(6655). On the surface of aCMOS image sensor, micro lens(6651) is formed for every pixel. Due toits lens properties, this micro lens takes the role of concentratingrays impinging on the CMOS image sensor toward the photodiode(6656).Under the micro lens lies a color filter(6652) which lets through onlyone color among the three primary colors. The color of the correspondingpixel is determined by this color filter. On the other hand, ashield(6653) is formed under the color filter in order to prevent crosstalks between pixels. Rays that passed through the micro lens and thecolor filter reaches the photodiode by the open area in the shield andtransformed into electrical charge. Furthermore, electrical wires(6654)are formed to draw out electrical current which has been produced byphotoelectrical conversion process on the surface of the photodiode.

Considering the structures of ordinary CMOS image sensors, it can beseen that insensitive pixels can be produced by various methods. First,if the photodiode(PD) is not formed or the area of the photodiode ismade very small, then photoelectrical conversion is not taking place, orits rate will be very small and practically there will be no videosignals. Furthermore, the shield can completely cover the photodiode, orthe open area can be made very tiny so that the amount of rays reachingthe photodiode is very small. Furthermore, instead of forming a colorfilter, forming a dark absorbing filter or a reflecting filter canaccomplish the same effect. Furthermore, instead of the micro lens whichconcentrates light rays, the micro lens can be formed into a concavelens so that it spreads light rays. Also, by not forming a micro lens,the amount of light rays reaching the photodiode can be reduced. Besidethese, it can be realized that diverse methods can be used to forminsensitive pixels in a CMOS image sensor.

In FIG. 65, the locations of sensitive pixels are determined in amathematically precise manner considering the real projection scheme ofthe fisheye lens that is used as a set and the projection scheme of thedesirable complex image. However, insensitive pixels do not havecorresponding object points in the world coordinate system, andtherefore, the locations of insensitive pixels can be arbitrarilydetermined. Practically, the locations of insensitive pixels can bedetermined considering the CMOS image sensor fabrication processes. Inthis case, the insensitive pixels must be close to the sensitive pixelssharing row or column numbers, and at the same time, insensitive pixelsmust not be too far apart or too close to each other among them.

Unlike bad pixels, such insensitive pixels are formed to havedeliberately determined patterns on the image display means. While FIG.48 illustrates a case where dormant pixels have been formed to formrectangular areas in the lower left and the lower right corners whereimages do not exist, FIG. 65 illustrates a case where insensitive pixelshave been formed to form rectangular areas in the lower left and thelower right corners where images do not exist. On the other hand, FIG.67 illustrates a case where insensitive pixels have been formed alongthe boundary planes between neighboring sub image planes in order todistinguish each sub image plane in multiple viewpoint panoramic images.In this case, multiple viewpoint panoramic images such as shown in FIG.26 can be obtained without any separate image processing, and darkboundary lines along the longitudinal direction are formed byinsensitive pixels. On the other hand, FIG. 68 is a case whereadditional insensitive pixels have been formed in order to distinguisheach sub image plane as in FIG. 49.

INDUSTRIAL APPLICABILITY

Such CMOS image sensors can be used not only in security•surveillanceapplications for indoor and outdoor environments, but also in diverseareas such as video phone for apartment entrance door, rear view camerafor automobiles, and visual sensors for robots.

The invention claimed is:
 1. A complex image acquisition devicecomprising: an image acquisition means for acquiring an uncorrectedimage plane using a camera equipped with a wide-angle imaging lensrotationally symmetric about an optical axis, an image processing meansfor extracting a processed image plane from the uncorrected image plane;and an image display means for displaying the processed image plane,wherein; the uncorrected image plane is a two-dimensional array withK_(max) rows and L_(max) columns, the processed image plane is atwo-dimensional array with I_(max) rows and J_(max) columns, a signalvalue of a live pixel with a pixel coordinate (I, J) on the processedimage plane and having a video signal is given as a signal value of animaginary image point on the uncorrected image plane due to an incidentray originating from an object point on an imaginary object plane in aworld coordinate system having a coordinate (X_(I,J), Y_(I,J),Z_(I,J))≡(X(I, J), Y(I, J), Z(I, J)), wherein; the said world coordinatesystem takes a nodal point of the said wide-angle lens as an origin andtakes the said optical axis as a Z-axis, wherein a direction from thesaid origin to the object plane is the positive(+) direction, an axispassing through the said origin and parallel to the sides of the imagesensor plane of the said camera along the longitudinal direction is aY-axis, wherein a direction from a top end of the image sensor plane toa bottom end is a positive(+) Y-axis direction, and the said worldcoordinate system is a right-handed coordinate system, the X coordinateof the said object point is given as a sole function of the lateralpixel coordinate J as in the following equation,X _(I,J) ≡X(I,J)=X(J)≡X _(J) and the Y coordinate of the said objectpoint is given as a sole function of the longitudinal pixel coordinate Ias in the following equation,Y _(I,J) |Y(I,J)=Y(I)≡Y _(I).
 2. The complex image acquisition device ofclaim 1, wherein; a Z coordinate of any one object point among the saidobject points is given as a sole function of the lateral pixelcoordinate J as in the following equation,Z _(I,J) ≡Z(I,J)=Z(J)≡Z _(j) or, a Z coordinate is given as a solefunction of the longitudinal pixel coordinate I as in the followingequation,Z _(I,J) ≡Z(I,J)=Z(I)≡Z _(I).
 3. The complex image acquisition device ofclaim 1, wherein; a lateral incidence angle of the said incident raygiven by the following equation is given as a monotonic function of thelateral pixel coordinate J,$\psi_{I,J} = {\tan^{- 1}\left( \frac{X_{J}}{Z_{I,J}} \right)}$ and alongitudinal incidence angle of the said incident ray given by thefollowing equation is given as a monotonic function of the longitudinalpixel coordinate I,$\delta_{I,J} = {{\tan^{- 1}\left( \frac{Y_{I}}{\sqrt{X_{J}^{2} + Z_{I,J}^{2}}} \right)}.}$4. The complex image acquisition device of claim 1, wherein; a pixelcoordinate of the optical axis on the said uncorrected image plane is(K_(o), L_(o)), a pixel coordinate of the optical axis on the saidprocessed image plane is (I_(o), J_(o)), a pixel coordinate of the saidimaginary image point is (x′_(I,J),y′_(I,J)), wherein; the said pixelcoordinate is given by the following equations,$\phi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I}}{X_{J}} \right)}$$\theta_{I,J} = {\cos^{- 1}\left( \frac{Z_{I,J}}{\sqrt{X_{J}^{2} + Y_{I}^{2} + Z_{I,J}^{2}}} \right)}$r_(I, J) = r(θ_(I, J)) x_(I, J)^(′) = L_(o) + gr_(I, J)cos  ϕ_(I, J)y_(I, J)^(′) = K_(o) + gr_(I, J)sin  ϕ_(I, J) a real projection schemeof the said lens is an image height r given as a function of the zenithangle θ of the corresponding incident ray and given as r=r(θ), said g isa camera magnification ratio and given as ${g = \frac{r^{\prime}}{r}},$here, r′ is a pixel distance on the uncorrected image planecorresponding to the image height r.
 5. The complex image acquisitiondevice of claim 1, wherein; the said object plane satisfies thefollowing equations on differentiable regions,${\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial J} \right)^{2}} = 1$${\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial I} \right)^{2}} = 1.$6. The complex image acquisition device of claim 1, wherein; the Zcoordinate of the said object point is a sole function of the lateralpixel coordinate J and given by the following equation,Z _(I,J) ≡Z(I,J)=Z(J)≡Z _(J).
 7. The complex image acquisition device ofclaim 6, wherein; the said object plane satisfies the following equationon differentiable regions,${\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{J}}{\partial J} \right)^{2}} = 1.$8. The complex image acquisition device of claim 7, wherein; the saidobject plane satisfies the following equation on differentiable regions,$\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} = 1.$
 9. Thecomplex image acquisition device of claim 1, wherein; the Z coordinateof the said object point is a sole function of the longitudinal pixelcoordinate I and given by the following equation,Z _(I,J) ≡Z(I,J)=Z(I)≡Z _(I).
 10. The complex image acquisition deviceof claim 9, wherein; the said object plane satisfies the followingequation on differentiable regions.${\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I}}{\partial I} \right)^{2}} = 1.$11. The complex image acquisition device of claim 10, wherein; the saidobject plane satisfies the following equation on differentiable regions,$\left( \frac{\partial X_{J}}{\partial J} \right)^{2} = 1.$
 12. Acomplex image acquisition device comprising: an image acquisition meansfor acquiring an uncorrected image plane using a camera equipped with awide-angle imaging lens rotationally symmetric about an optical axis, animage processing means for extracting a processed image plane from theuncorrected image plane; and an image display means for displaying theprocessed image plane, wherein; the said uncorrected image plane is atwo-dimensional array with K_(max) rows and L_(max) columns, the saidprocessed image plane is a two-dimensional array with I_(max) rows andJ_(max) columns, a signal value of a live pixel with a pixel coordinate(I, J) on the said processed image plane and having a video signal isgiven as a signal value of an imaginary image point having a pixelcoordinate (x′_(I,J),y′_(I,J)) on the uncorrected image plane due to anincident ray originating from an object point on an imaginary objectplane in a world coordinate system, the said world coordinate systemtakes a nodal point of the said wide-angle lens as an origin and takesthe said optical axis as a Z-axis, wherein a direction from the saidorigin to the object plane is the positive(+) direction, an axis passingthrough the origin and parallel to the sides of the image sensor planeof the said camera along the longitudinal direction is a Y-axis, whereina direction from the top end of the image sensor plane to the bottom endis the positive(+) Y-axis direction, and the said world coordinatesystem is a right-handed coordinate system, a zenith angle of the saidincident ray is given as θ_(I,J)=θ(I,J), an azimuth angle of the saidincident ray is given as φ_(I,J)=φ(I,J), a pixel coordinate of theoptical axis on the said uncorrected image plane is (K_(o), L_(o)), apixel coordinate of the said imaginary image point is given by thefollowing equation,x′ _(I,J) =L _(o) +gr(θ_(I,J))cos φ_(I,J)y′ _(I,J) =K _(o) +gr(θ_(I,J))sin φ_(I,J) a real projection scheme ofthe said lens is an image height r given as a function of the zenithangle θ of the corresponding incident ray and given as r=r(θ), the saidg is a camera magnification ratio and given as${g = \frac{r^{\prime}}{r}},$ here, r′ is a pixel distance on theuncorrected image plane corresponding to the image height r.
 13. Thecomplex image acquisition device of claim 12, wherein; a pixelcoordinate of the optical axis on the said processed image plane is(I_(o), J_(o)), a zenith angle of the said incident ray is given by thefollowing equation,${\theta\left( {I,J} \right)} = {2{\tan^{- 1}\left\lbrack {\frac{\tan\left( \frac{\theta_{2}}{2} \right)}{r_{2}^{''}}r_{I,J}^{''}} \right\rbrack}}$here, a pixel distance corresponding to a pixel coordinate (I, J) on theprocessed image plane is given by the following equation,r″ _(I,J)=√{square root over ((I−I _(o))²+(J−J _(o))²)}{square root over((I−I _(o))²+(J−J _(o))²)} θ₂ is a reference zenith angle of an incidentray, r″₂ is a pixel distance on the processed image plane correspondingto the said reference zenith angle θ₂, and the azimuth angle of the saidincident ray is given by the following equation,${\phi\left( {I,J} \right)} = {{\tan^{- 1}\left( \frac{I - I_{o}}{J - J_{o}} \right)}.}$14. The complex image acquisition device of claim 12, wherein; a pixelcoordinate of the optical axis on the said processed image plane is(I_(o), J_(o)), a zenith angle of the said incident ray is given by thefollowing equation,${\theta\left( {I,J} \right)} = {\frac{\theta_{2}}{r_{2}^{''}}r_{I,J}^{''}}$here, a pixel distance corresponding to a pixel coordinate (I, J) on theprocessed image plane is given by the following equation,r″ _(I,J)=√{square root over ((I−I _(o))²+(J−J _(o))²)}{square root over((I−I _(o))²+(J−J _(o))²)} θ₂ is a reference zenith angle of an incidentray, r″₂ is a pixel distance on the processed image plane correspondingto the said reference zenith angle θ₂, and the azimuth angle of the saidincident ray is given by the following equation,${\phi\left( {I,J} \right)} = {{\tan^{- 1}\left( \frac{I - I_{o}}{J - J_{o}} \right)}.}$15. A complex image acquisition device comprising: an image acquisitionmeans for acquiring an uncorrected image plane using a camera equippedwith a wide-angle imaging lens rotationally symmetric about an opticalaxis, an image processing means for extracting a processed image planefrom the uncorrected image plane; and an image display means fordisplaying the processed image plane, wherein; the said uncorrectedimage plane is a two-dimensional array with K_(max) rows and L_(max)columns, the said processed image plane is a two-dimensional array withI_(max) rows and J_(max) columns, a signal value of a live pixel with apixel coordinate (I, J) on the said processed image plane and having avideo signal is given as a signal value of an imaginary image pointhaving a pixel coordinate (x′_(I,J),y′_(I,J)) on the uncorrected imageplane due to an incident ray originating from an object point on animaginary object plane in a world coordinate system, the said worldcoordinate system takes a nodal point of the said wide-angle lens as anorigin and takes the said optical axis as a Z-axis, wherein a directionfrom the said origin to the object plane is the positive(+) direction,an axis passing through the origin and parallel to the sides of theimage sensor plane of the said camera along the longitudinal directionis a Y-axis, wherein a direction from the top end of the image sensorplane to the bottom end is the positive(+) Y-axis direction, and thesaid world coordinate system is a right-handed coordinate system, alateral incidence angle of the said incident ray is given as a solefunction of the lateral pixel coordinate J and given as ψ_(J)≡ψ(J), alongitudinal incidence angle of the said incident ray is given as a solefunction of the longitudinal pixel coordinate I and given as δ_(I)≡δ(I),a pixel coordinate of the optical axis on the said uncorrected imageplane is (K_(o), L_(o)), a pixel coordinate of the said imaginary imagepoint is given by the following equation,$\phi_{I,J} = {\tan^{- 1}\left( \frac{\tan\;\delta_{I}}{\sin\;\psi_{J}} \right)}$θ_(I, J) = cos⁻¹(cos  δ_(I)cos  ψ_(J))x_(I, J)^(′) = L_(o) + g r(θ_(I, J))cos  ϕ_(I, J)y_(I, J)^(′) = K_(o) + g r(θ_(I, J))sin  ϕ_(I, J) a real projectionscheme of the said lens is an image height r given as a function of thezenith angle θ of the corresponding incident ray and given as r=r(θ),the said g is a camera magnification ratio and given as${g = \frac{r^{\prime}}{r}},$ here, r′ is a pixel distance on theuncorrected image plane corresponding to the image height r.
 16. A CMOSimage sensor that is used as a set with a wide-angle imaging lensrotationally symmetric about an optical axis, wherein; the said CMOSimage sensor has a multitude of pixels in an image sensor plane given ina form of an array having I_(max) rows and J_(max) columns, the saidmultitude of pixels contain at least one sensitive pixel having a videosignal, an intersection point between the image sensor plane and theoptical axis is a reference point O having a row number I_(o) and acolumn number J_(o), the said row number I_(o) is a real number largerthan 1 and smaller than I_(max), the said column number J_(o) is a realnumber larger than 1 and smaller than J_(max), in a first rectangularcoordinate system, wherein; an axis that passes through the saidreference point and is parallel to the sides of the image sensor planealong the lateral direction is taken as an x-axis, and the positivex-axis runs from the left side of the said image sensor plane to theright side, an axis that passes through the said reference point and isparallel to the sides of the image sensor plane along the longitudinaldirection is taken as an y-axis, and the positive y-axis runs from thetop end of the said image sensor plane to the bottom end, a centercoordinate of a sensitive pixel P(I, J) with a row number I and a columnnumber J and having a video signal is given as (x_(I,J), y_(I,J)),wherein; the center coordinate (x_(I,J), y_(I,J)) of the said sensitivepixel P(I, J) is given as a coordinate of an imaginary image point dueto an incident ray originating from an object point on an imaginaryobject plane in a world coordinate system having a coordinate (X_(I,J),Y_(I,J), Z_(I,J))≡(X(I, J), Y(I, J), Z(I, J)), the said world coordinatesystem takes a nodal point of the said wide-angle lens as an origin andtakes the said optical axis as a Z-axis, wherein a direction from thesaid origin to the object plane is the positive(+) direction, an axispassing through the origin and is parallel to the sides of the imagesensor plane of the said camera along the longitudinal direction is aY-axis, wherein a direction from the top end of the image sensor planeto the bottom end is the positive(+) Y-axis direction, and the saidworld coordinate system is a right-handed coordinate system, the Xcoordinate of the said object point is given as a sole function of thelateral pixel coordinates J as in the following equation,X _(I,J) ≡X(I,J)=X(J)≡X _(j) and the Y coordinate of the said objectpoint is given as a sole function of the longitudinal pixel coordinate Ias in the following equation,Y _(I,J) ≡Y(I,J)=Y(I)≡Y _(I).
 17. The CMOS image sensor of claim 16,wherein; a Z coordinate of any one object point among the said objectpoints is given as a sole function of the lateral pixel coordinate J asin the following equation,Z _(I,J) ≡Z(I,J)=Z(J)≡Z _(J) or, the Z coordinate is given as a solefunction of the longitudinal pixel coordinate I as in the followingequation,Z _(I,J) ≡Z(I,J)=Z(I)≡Z _(I).
 18. The CMOS image sensor of claim 16,wherein; a lateral incidence angle of the said incident ray given by thefollowing equation is given as a monotonic function of the lateral pixelcoordinates J,$\psi_{I,J} = {\tan^{- 1}\left( \frac{X_{J}}{Z_{I,J}} \right)}$ and alongitudinal incidence angle of the said incident ray given by thefollowing equation is given as a monotonic function of the longitudinalpixel coordinate I,$\delta_{I,J} = {{\tan^{- 1}\left( \frac{Y_{I}}{\sqrt{X_{J}^{2} + Z_{I,J}^{2}}} \right)}.}$19. The CMOS image sensor of claim 16, wherein; a center coordinate ofthe said sensitive pixel is (x_(I,J),y_(I,J)), wherein; the said centercoordinate is given by the following equations,$\phi_{I,J} = {\tan^{- 1}\left( \frac{Y_{I}}{X_{J}} \right)}$$\theta_{I,J} = {\cos^{- 1}\left( \frac{Z_{I,J}}{\sqrt{X_{J}^{2} + Y_{I}^{2} + Z_{I,J}^{2}}} \right)}$r_(I, J) = r(θ_(I, J)) x_(I, J) = r_(I, J)cos  ϕ_(I, J)y_(I, J) = r_(I, J)sin  ϕ_(I, J) here, r=r(θ) is a real projectionscheme of the said lens and is an image height r given as a function ofthe zenith angle θ of the corresponding incident ray.
 20. The CMOS imagesensor of claim 16, wherein; the said object plane satisfies thefollowing equations on differentiable regions,${\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial J} \right)^{2}} = 1$${\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I,J}}{\partial I} \right)^{2}} = 1.$21. The CMOS image sensor of claim 16, wherein; the Z coordinate of thesaid object point is a sole function of the lateral pixel coordinate Jand given by the following equation,Z _(I,J) ≡Z(I,J)=Z(J)≡Z _(J).
 22. The CMOS image sensor of claim 21,wherein; the said object plane satisfies the following equation ondifferentiable regions,${\left( \frac{\partial X_{J}}{\partial J} \right)^{2} + \left( \frac{\partial Z_{J}}{\partial J} \right)^{2}} = 1.$23. The CMOS image sensor of claim 22, wherein; the said object planesatisfies the following equation on differentiable regions,$\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} = 1.$
 24. The CMOSimage sensor of claim 16, wherein; the Z coordinate of the said objectpoint is a sole function of the longitudinal pixel coordinate I andgiven by the following equation,Z _(I,J) ≡Z(I,J)=Z(I)≡Z _(I).
 25. The CMOS image sensor of claim 24,wherein; the said object plane satisfies the following equation ondifferentiable regions,${\left( \frac{\partial Y_{I}}{\partial I} \right)^{2} + \left( \frac{\partial Z_{I}}{\partial I} \right)^{2}} = 1.$26. The CMOS image sensor of claim 25, wherein; the said object planesatisfies the following equation on differentiable regions.$\left( \frac{\partial X_{J}}{\partial J} \right)^{2} = 1.$
 27. A CMOSimage sensor that is used as a set with a wide-angle imaging lensrotationally symmetric about an optical axis, wherein; the said CMOSimage sensor has a multitude of pixels in an image sensor plane given ina form of an array having I_(max) rows and J_(max) columns, the saidmultitude of pixels contain at least one sensitive pixel having a videosignal, an intersection point between the image sensor plane and theoptical axis is a reference point O having a row number I_(o) and acolumn number J_(o), the said row number I_(o) is a real number largerthan 1 and smaller than I_(max), the said column number J_(o) is a realnumber larger than 1 and smaller than J_(max), in a first rectangularcoordinate system, wherein; an axis that passes through the saidreference point and is parallel to the sides of the image sensor planealong the lateral direction is taken as an x-axis, and the positivex-axis runs from the left side of the said image sensor plane to theright side, an axis that passes through the said reference point and isparallel to the sides of the image sensor plane along the longitudinaldirection is taken as an y-axis, and the positive y-axis runs from thetop end of the said image sensor plane to the bottom end, a centercoordinate (x_(I,J), y_(I,J)) of a sensitive pixel P(I, J) with a rownumber I and a column number J and having a video signal is given as acoordinate of an imaginary image point due to an incident rayoriginating from an object point on an imaginary object plane, wherein;a zenith angle of the said incident ray is given as θ_(I,J)=θ(I,J) andan azimuth angle is given as φ_(I,J)=φ(I,J), the center coordinate ofthe said sensitive pixel is given by the following equation,x _(I,J) =r(θ_(I,J))cos φ_(I,J)y _(I,J) =r(θ_(I,J))sin φ_(I,J) here, r=r(θ) is a real projection schemeof the said lens and is an image height r given as a function of thezenith angle θ of the corresponding incident ray.
 28. The CMOS imagesensor of claim 27, wherein; the zenith angle of the said incident rayis given by the following equation,${\theta\left( {I,J} \right)} = {2{\tan^{- 1}\left\lbrack {\frac{\tan\left( \frac{\theta_{2}}{2} \right)}{r_{2}^{''}}r_{I,J}^{''}} \right\rbrack}}$here, a pixel distance corresponding to the pixel coordinate (I, J) isgiven by the following equation,r″ _(I,J)=√{square root over ((I−I _(o))²+(J−J _(o))²)}{square root over((I−I _(o))²+(J−J _(o))²)} θ₂ is a reference zenith angle of an incidentray, r″₂ is a pixel distance corresponding to the said reference zenithangle θ₂, and the azimuth angle of the said incident ray is given by thefollowing equation,${\phi\left( {I,J} \right)} = {{\tan^{- 1}\left( \frac{I - I_{o}}{J - J_{o\;}} \right)}.}$29. The CMOS image sensor of claim 27, wherein; the zenith angle of thesaid incident ray is given by the following equation,${\theta\left( {I,J} \right)} = {\frac{\theta_{2}}{r_{2}^{''}}r_{I,J}^{''}}$here, a pixel distance corresponding to the pixel coordinate (I, J) isgiven by the following equation,r″ _(I,J)√{square root over ((I−I _(o))²+(J−J _(o))²)}{square root over((I−I _(o))²+(J−J _(o))²)} θ₂ is a reference zenith angle of an incidentray, r″₂ is a pixel distance corresponding to the said reference zenithangle θ₂, and the azimuth angle of the said incident ray is given by thefollowing equation,${\phi\left( {I,J} \right)} = {{\tan^{- 1}\left( \frac{I - I_{o}}{J - J_{o\;}} \right)}.}$30. A CMOS image sensor that is used as a set with a wide-angle imaginglens rotationally symmetric about an optical axis, wherein; the saidCMOS image sensor has a multitude of pixels in an image sensor planegiven in a form of an array having I_(max) rows and J_(max) columns, thesaid multitude of pixels contain at least one sensitive pixel having avideo signal, an intersection point between the image sensor plane andthe optical axis is a reference point O having a row number I_(o) and acolumn number J_(o), the said row number I_(o) is a real number largerthan 1 and smaller than I_(max), the said column number J_(o) is a realnumber larger than 1 and smaller than J_(max), in a first rectangularcoordinate system, wherein; an axis that passes through the saidreference point and is parallel to the sides of the image sensor planealong the lateral direction is taken as an x-axis, and the positivex-axis runs from the left side of the said image sensor plane to theright side, an axis that passes through the said reference point and isparallel to the sides of the image sensor plane along the longitudinaldirection is taken as an y-axis, and the positive y-axis runs from thetop end of the said image sensor plane to the bottom end, a centercoordinate (x_(I,J), y_(I,J)) of a sensitive pixel P(I, J) with a rownumber I and a column number J and having a video signal is given as acoordinate of an imaginary image point due to an incident rayoriginating from an object point on an imaginary object plane, wherein;a lateral incidence angle of the said incidence ray is given as a solefunction of the lateral pixel coordinate J and is given as ψ_(J)≡ψ(J), alongitudinal incidence angle is given as a sole function of thelongitudinal pixel coordinate I and is given as δ_(I)≡δ(I), the centercoordinate of the said pixel is given by the following equations,$\phi_{I,J} = {\tan^{- 1}\left( \frac{\tan\;\delta_{I}}{\sin\;\psi_{J}} \right)}$θ_(I, J) = cos⁻¹(cos  δ_(I)cos  ψ_(J))x_(I, J) = r(θ_(I, J))cos  ϕ_(I, J)y_(I, J) = g r(θ_(I, J))sin  ϕ_(I, J) here, r=r(θ) is a real projectionscheme of the said lens and is an image height r given as a function ofthe zenith angle θ of the corresponding incident ray.